This is a tutorial for using the Python interface to the giftfile system. As a prerequisite, you should probably read "Giftpool Protocol Synopsis" (http://giftfile.org/documents/giftpool_protocol_synopsis) in order to get a general feel for how giftfile system transactions work.
The tutorial's code is presented as an interactive python session. Presumably you could start a Python shell (just type "python" on the command line) and follow along.
For those relatively new to Python, it's important to know that you can get documentation for any module from the command line (e.g., "pydoc giftfile", or "pydoc giftfile.clientlib").
All modules for the Giftfile Project are contained in a Python package called giftfile. We'll be using the clientlib module, which is the client interface to a giftpool. The transaction_status module contains codes we'll need to reference in order to check the status of responses from the server.
ISSUE: our module documentation is spotty
>>> from giftfile import clientlib, transaction_status
As use of the clientlib module will end up making command line calls to gnupg, and gnupg writes fairly verbose messages on stderr, we direct its output to a log file.
>>> clientlib.pgp_log_file = file('pgp.log', 'w')
We're going to start gathering the data required for opening a connection to a giftpool. First, we need to know the giftpool's URL.
>>> giftpool_url = 'http://test-giftpool.giftfile.org/'
Optionally, we may also make an identity object for the giftpool. This represents our assertion of exactly who it is we're communicating with. The identity is used to verify the digital signature contained in all responses from the server. Should the local identity not match the real identity of the server, an exception will be raised. It's strongly recommended that a giftpool identity be supplied when opening a server connection, otherwise your communications will be susceptible to various attacks.
To use the giftpool identity feature, you must have the the giftpool's PGP key in your keyring and know the fingerprint of that key. Ideally, the giftpool's key will not only be in your keyring, but will have sufficient trust assigned to it by way of PGP's web of trust.
ISSUE: Currently the library does not provide information as to whether the giftpool's key is trusted or not.
>>> #giftpool_id = None # Not recommended! >>> giftpool_fingerprint = '44f91abd6dc120cf8a33e11459a6d055e23478ce' >>> giftpool_id = clientlib.Identity(giftpool_fingerprint)
Optionally, we prepare an object for caching HTTP responses from the server. It's important to supply a cache object because it enables the ability to poll server records efficiently. The caching that is performed is transparent, meaning we don't need to be aware that it's happening. It's also synchronous with the state of the server, meaning we don't have to worry about seeing stale data.
The clientlib module provides a simple cache implementation that is memory resident. It has features such as the ability to purge entries older than a certain date, purge old entries until memory use is less than some amount, and save and restore its state to a file. If this implementation isn't suitable, you can supply your own.
>>> http_cache = clientlib.MemoryHttpCache()
Given the URL and identity of the giftpool, and the cache object, we can finally open a connection. The connection will use the persistence feature of HTTP, allowing efficient transfer of multiple messages to and from the server. Should the connection time out during the session, it will be transparently reestablished at the time of the next request.
>>> giftpool = clientlib.GiftpoolInterface(giftpool_url, giftpool_id, ... http_cache)
We'll start using the giftpool connection by accessing the info record, which contains various data about the giftpool.
clientlib.GiftpoolInterface exposes the fact that giftpool protocol messages are in XML format. The interface provides access to the XML of server responses in various forms, including text, as a libxml2 document, and as an xml2tramp object (see giftfile.xml2tramp). Here, we'll use a simple XML object called an element map, where XML elements and their values are presented as a recursive Python dictionary (see giftfile._john_xml.ElementMap).
>>> info_record = giftpool.get_info().element_map
Next, we access some fields of the info record. Note that the donation_friction field, a decimal value, will be returned as a string. clientlib.GiftpoolInterface does not provide any conversion to native Python types. Later, a giftpool interface that is more native to Python will be presented.
>>> info_record['name'] 'Giftfile Project Test Giftpool' >>> info_record['issue_uri'] 'mailto:test-giftpool-issues@giftfile.org' >>> info_record['user_keyserver'] 'http://test-giftpool.giftfile.org:11372/' >>> info_record['currency_unit'] 'USD' >>> info_record['donation_friction'] '0.25'
Now we'd like to start our first transaction with a giftpool: a donation.
To conduct a transaction, you'll need to have your own PGP key. On top of that, your key must be uploaded to the giftpool's keyserver, which is given in the user_keyserver field we have just inspected. Creation and management of PGP keys is outside the scope of this interface.
We begin by making an identity object which represents us.
>>> user_fingerprint = 'bd408e29a8ffe9e6f2b2336bedc58449f3c4c9a0' >>> user_id = clientlib.Identity(user_fingerprint)
The first part of a donation transaction is making a donation request to the giftpool. Here, "donation request" doesn't mean that we're requesting a donation from the giftpool, but rather that we are requesting to make a donation to the giftpool.
To make a request to the giftpool, we first create a request object of the appropriate type. Donation requests are very simple, and we only need to provide our identity and the time the request was issued.
>>> donation_request = giftpool.create_donation(user_id)
The reason we first create a request object, rather than create and send the request all in one operation, is related to the topic of reliable messaging. If our program crashes at some inopportune time, or we otherwise aren't sure if the server received our request, we must send the exact same request again. In the case that the server did receive the original request, it will simply ignore the duplicate.
In support of reliable messaging, we should really be saving the donation_request object to a reliable storage before making the request, but will forgo that here to keep the tutorial simple.
ISSUE: request objects should support pickling
Now it's time to send off our donation request. We are really only interested in one attribute of the response object, namely the URL of the resulting donation record.
>>> donation_url = donation_request.send().url >>> donation_url 'http://test-giftpool.giftfile.org/1/private/donations/364caf9341cfd317210fe ffb18eaacf5085575b2'
From the URL, we retrieve the donation record itself. We'll inspect the status field of the record, which indicates the progress of the transaction. We expect it to be "Payment not received", which means that the giftpool is awaiting our payment.
>>> donation_record = giftpool.get_donation(donation_url).element_map >>> donation_record['status'] '1.2 Payment Not Received' >>> assert transaction_status.get_code(donation_record['status']) == \ ... transaction_status.PAYMENT_NOT_RECEIVED
From here, for a real donation transaction, we would write a check to the giftpool and send it off in the mail. The payee name and address are obtained from the giftpool info record. Here is an excellent example of why you should use a verified giftpool identity when creating a giftpool connection. If you don't, a man in the middle is able to intercept your access of the giftpool info record and insert his own address.
A payment ID, which is included in the donation record, must be in the memo field of the check. The payment ID is used by the giftpool to associate our payment with the donation request just made.
>>> info_record['donation_payee'] 'EGE, Inc.' >>> print info_record['donation_address'] TEST!! Giftpool Donations EGE, Inc. PO BOX 392 GREENFIELD CENTER NY 12833-0392 USA >>> donation_record['payment_id'] '085575b2'
Now we would periodically poll the donation record, perhaps twice a day, to watch for a change in status. Obviously, the tracking of outstanding transactions requires a fair amount of resource management by the client. Such resource management will not be addressed in this simple tutorial.
One thing we can demonstrate, however, is the effectiveness of the transparent caching of server responses. Here we fetch the donation record again, and inspect the HTTP response object to show that it is indeed a cached response, as the record has not changed since our last access.
>>> http_response = giftpool.get_donation(donation_url).http_response >>> http_response.is_cached_response True
Since we've made at least one donation request, the giftpool holds a user record for our identity. As an exercise, let's access the user record and inspect a few fields. User records can be accessed by first making a query by the ID of the user, which is the same as his fingerprint. The query will yield the URL of the user record.
One of the fields we inspect is the list of donations associated with our identity. We expect to see the URL of our pending donation's record in the list.
>>> user_url = giftpool.locate_user_record(user_fingerprint).url >>> user_record = giftpool.get_user(user_url).element_map >>> user_record['status'] '1.1 New User' >>> user_record['allocation_privilege'] '0.00' >>> print user_record['donations'] {'li': 'http://test-giftpool.giftfile.org/1/private/donations/364caf9341cfd3 17210feffb18eaacf5085575b2'}
With the donations list, again we see a case where the data is not in a form natural to Python. Instead of a simple list of URL's, we are exposed to the XML representation of the list, where each item is contained in an <li> element. To alleviate this, let's switch gears to an interface more native to Python.
This other interface is provided as an alternative implementation of the GiftpoolInterface class, and is contained in the clientlib2 module. The main difference is in the server response objects, which in this case provide an abstraction from XML. In these objects, fields of a response record are accessed as Python attributes, and are of types native to Python.
ISSUE: the clientlib2 interface is incomplete and not well documented
After replacing our giftpool object with an instance of clientlib2.GiftpooInterface, let's take a look at the same user record fields that we inspected previously.
>>> from giftfile import clientlib2 >>> giftpool = clientlib2.GiftpoolInterface(giftpool_url, giftpool_id, ... http_cache) >>> user_record = giftpool.get_user(user_url) >>> user_record.status '1.1 New User' >>> user_record.status_code '1.1' >>> user_record.allocation_privilege FixedPoint('0.00', 2) >>> user_record.donations ['http://test-giftpool.giftfile.org/1/private/donations/364caf9341cfd317210f effb18eaacf5085575b2']
Of interest is status_code, which is a convenience attribute provided with all transaction records. Its value is equivalent to:
>>> transaction_status.get_code(user_record.status) '1.1'
Also note the use of the FixedPoint type (see giftfile.fixedpoint), which is used to represent decimal numbers exactly. FixedPoint values can be used just as any number type built into Python. Here is a comparison with the native floating point type.
>>> a = .1 >>> a *= 3 >>> a 0.30000000000000004 >>> print a 0.3 >>> from giftfile.fixedpoint import FixedPoint >>> b = FixedPoint('.1') >>> b *= 3 >>> b FixedPoint('0.30', 2) >>> print b 0.30
Now that we've got a more comfortable interface, let us return to the pending donation transaction.
It's time to face a problem. Normally, before a giftpool will allow a donation transaction to complete, it would have to accept delivery of the corresponding check, and clear that check with a bank. Since we are only practicing, how can we push our donation through so that we can try allocations and other types of giftpool transactions? The answer is that some giftpools are created for just this type of testing, allowing users to drive their own transactions.
Let's make sure that ours is a testing giftpool.
>>> info_record = giftpool.get_info() >>> assert info_record.user_testing_capability
Given a giftpool with user testing capability, we are able to play the role of administrator with our own transactions. Here, we make an update request, completing the donation transaction as if the giftpool had successfully processed a $25.00 payment from us.
>>> giftpool.create_donation_update(user_id, donation_url, is_received=True, ... amount=25.00, payer_name='Test Payer', check_number='101', ... is_cleared=True).send()
Now we look at how this action has affected the donation's record and our user record.
>>> donation_record = giftpool.get_donation(donation_url) >>> donation_record.status '2.0 OK' >>> donation_record.donation_amount, donation_record.privilege_amount (FixedPoint('25.00', 2), FixedPoint('24.75', 2)) >>> user_record = giftpool.get_user(user_url) >>> user_record.allocation_privilege FixedPoint('24.75', 2)
After transactions such as a donation are completed, we may delete the associated records.
>>> records_to_delete = [donation_url] >>> giftpool.create_record_deletion(user_id, records_to_delete).send() >>> user_record = giftpool.get_user(user_url) >>> user_record.donations []
With our donation completed, we can start making allocations to the giftfiles of our choice.
To allocate to a giftfile, we need its certificate. The current giftpool server implementation cannot deal with real giftfile certificates yet. Instead, it uses a simple placeholder containing only three attributes: the ID (PGP fingerprint) of the publisher, the cryptographic digest of the associated work, and the cryptographic digest of the metadata itself. (For a glimpse of real certificates, see the giftfile.certificate module.)
With giftfiles come the notion of publishers. To continue with our test, a second PGP keypair will come in handy for use as a publisher identity. It is possible to use the same identity that was used for the donation, but it would be a little strange for one to make allocations to his own giftfiles. Using a separate identity lets us easily distinguish the roles of supporter and producer. Each role will have a separate user record in the giftpool.
>>> publisher_fingerprint = 'c6cc9b5e5173c8ff5b186a87d6c8a533b88f69f1' >>> publisher_id = clientlib.Identity(publisher_fingerprint)
Using this publisher identity, let's prepare a couple of mock giftfile certificates for use in allocation testing. For the purposes of this test, we can use random values for the cryptographic digests.
>>> from random import randrange >>> def get_random_digest(): ... return '%040x' % randrange(0, 2**160) ... >>> from giftfile.clientlib import Certificate >>> certificate_1 = Certificate( ... publisher_id = publisher_id, ... work_digest = get_random_digest(), ... message_digest = get_random_digest()) >>> certificate_2 = Certificate( ... publisher_id = publisher_id, ... work_digest = get_random_digest(), ... message_digest = get_random_digest())
The certificate message digest is also known an the giftfile ID. Giftfile ID's become important in several situations. One is when we have a certificate and would like to look up a certain giftpool's record of the corresponding giftfile. We'll see an example of that later. The other situation where giftfile ID's are vital is in allocation requests. Since certificates (at least, real certificates) can be verbose, normally an allocation request references a giftfile by ID rather than include the entire certificate.
Here we attempt to make a $0.50 allocation to the giftfile corresponding to certificate_1.
>>> allocations = [(certificate_1.get_reference(), 0.50)] >>> allocation_url = giftpool.create_allocation(user_id, ... allocations).send().url
When creating the allocation request, a list of allocations is provided. This is because allocations to multiple giftfiles may be grouped into a single request. Each allocation in the group is allowed to succeed or fail independently of the others, so a status is associated with each, in addition to an overall status for the entire transaction.
As we see here, the overall status only indicates whether all allocations contained in the request have finished processing, regardless of their individual success or failure. It's important to note that while certain testing giftpools may process allocation requests immediately, normally there is some delay which causes these transactions to be nondeterministic.
>>> allocation_record = giftpool.get_allocation(allocation_url) >>> allocation_record.status '2.0 OK' >>> allocation_record.total_requested_amount FixedPoint('0.50', 2) >>> allocation_record.total_successful_amount FixedPoint('0.00', 2) >>> allocation_record.allocations[0].status '4.5 Invalid Resource'
So why did our $0.50 allocation fail? The reason is that the giftpool does not have the certificate we are referring to in its records. Since we had just created the certificate using random digests, this result is not surprising. However, even under more realistic circumstances, such an exception will sometimes occur. When it does, we must put the failed allocations in a new request, this time including the full certificate rather than just a reference:
>>> allocations = [(certificate_1, 0.50)] >>> allocation_url = giftpool.create_allocation(user_id, ... allocations).send().url >>> allocation_record = giftpool.get_allocation(allocation_url) >>> allocation_record.status '2.0 OK' >>> allocation_record.total_requested_amount FixedPoint('0.50', 2) >>> allocation_record.total_successful_amount FixedPoint('0.50', 2) >>> allocation_record.allocations[0].status '2.0 OK'
Let's try one more allocation request, this time containing an allocation to each of our mock giftfiles. We'll use a reference for both certificates. The giftpool has already seen the full certificate for one of the giftfiles, so we expect that allocation to succeed. For the other, the situation is the same as before: since the giftpool does not have that giftfile's certificate on record, the allocation will fail.
>>> allocations = [ ... (certificate_1.get_reference(), 16.00), ... (certificate_2.get_reference(), 0.75), ... ] >>> allocation_url = giftpool.create_allocation(user_id, ... allocations).send().url >>> allocation_record = giftpool.get_allocation(allocation_url) >>> allocation_record.status '2.0 OK' >>> allocation_record.total_requested_amount FixedPoint('16.75', 2) >>> allocation_record.total_successful_amount FixedPoint('16.00', 2) >>> [item.status for item in allocation_record.allocations] ['2.0 OK', '4.5 Invalid Resource']
Let's check our user record again to see how these transactions have depleted our allocation privilege.
>>> user_record = giftpool.get_user(user_url) >>> user_record.total_allocation_amount FixedPoint('16.50', 2) >>> user_record.allocation_privilege FixedPoint('8.25', 2)
Now let's switch to the role of the publisher, and check the giftpool's records for our giftfiles. This query works much the same way as the user record query we made early in the tutorial.
>>> giftfile_1_url = giftpool.locate_giftfile_record( ... certificate_1.get_id()).url >>> giftfile_2_url = giftpool.locate_giftfile_record( ... certificate_2.get_id()).url Traceback (most recent call last): ... UnexpectedHttpResponse: 404 Not Found (expected 302) <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"> <html> <head><title>Error: 404 Not Found</title></head> <body> <h1>Error: 404 Not Found</h1> <pre>Invalid resource "giftfiles/891edbf34de3122287390342758ea8b4d3 fc53a5"</pre> </body> </html>
Not surprisingly, there was a failure in the query corresponding to the certificate which the giftpool does not have on record.
ISSUE: the server error response should not be in HTML format
Now we grab the record of the giftfile that is known by the giftpool:
>>> giftfile_record = giftpool.get_giftfile(giftfile_1_url) >>> giftfile_record.allocations_count 2 >>> giftfile_record.allocations_amount FixedPoint('16.50', 2)
In the role of publisher, let's inspect our user record. We expect the income to our giftfiles to raise both our allocation and remittance privileges.
>>> user_url = giftpool.locate_user_record(publisher_fingerprint).url >>> user_record = giftpool.get_user(user_url) >>> user_record.giftfiles ['http://test-giftpool.giftfile.org/1/private/giftfiles/c09361be95a813a925b3 ebb5c7165e314acc90b9'] >>> user_record.total_income_amount FixedPoint('16.50', 2) >>> user_record.allocation_privilege FixedPoint('15.50', 2) >>> user_record.remittance_privilege FixedPoint('15.50', 2)
It's time to try a remittance transaction. This transaction will proceed similar to the donation transaction we conducted earlier. We'll make an an initial request, and then play the role of administrator to push our transaction through.
>>> payee_address = """\ ... Test Payee ... 555 Main St ... New York NY 55555""" >>> remittance_amount = user_record.remittance_privilege >>> remittance_url = giftpool.create_remittance(publisher_id, ... remittance_amount, 'Test Payee', payee_address).send().url >>> remittance_record = giftpool.get_remittance(remittance_url) >>> remittance_record.status '1.0 Processing' >>> remittance_record.remittance_amount FixedPoint('15.50', 2) >>> remittance_record.privilege_amount FixedPoint('16.50', 2) >>> from giftfile.time_util import get_date >>> giftpool.create_remittance_update(publisher_id, remittance_url, ... is_issued=True, check_number='523', check_date=get_date()).send() >>> remittance_record = giftpool.get_remittance(remittance_url) >>> remittance_record.status '1.4 Remittance Issued'
Finally, one last look at the publisher's user record, to see the effects of the remittance.
>>> user_record = giftpool.get_user(user_url) >>> user_record.remittances ['http://test-giftpool.giftfile.org/1/private/remittances/0c6a0939e389473062 ff991263cf8a20ff82db24'] >>> user_record.total_remittance_amount FixedPoint('15.50', 2) >>> user_record.allocation_privilege FixedPoint('0.00', 2) >>> user_record.remittance_privilege FixedPoint('0.00', 2)
$LastChangedDate: 2004-05-25 16:58:17 -0400 (Tue, 25 May 2004) $