Planeta PythonBrasil

sourmash 3 was released last week, finally landing the Rust backend. But, what changes when developing new features in sourmash? I was thinking about how to best document this process, and since PR #826 is a short example touching all the layers I decided to do a small walkthrough.

Shall we?

The problem

The first step is describing the problem, and trying to convince reviewers (and yourself) that the changes bring enough benefits to justify a merge. This is the description I put in the PR:

Calling .add_hash() on a MinHash sketch is fine, but if you're calling it all the time it's better to pass a list of hashes and call .add_many() instead. Before this PR add_many just called add_hash for each hash it was passed, but now it will pass the full list to Rust (and that's way faster).

No changes for public APIs, and I changed the _signatures method in LCA to accumulate hashes for each sig first, and then set them all at once. This is way faster, but might use more intermediate memory (I'll evaluate this now).

There are many details that sound like jargon for someone not familiar with the codebase, but if I write something too long I'll probably be wasting the reviewers time too. The benefit of a very detailed description is extending the knowledge for other people (not necessarily the maintainers), but that also takes effort that might be better allocated to solve other problems. Or, more realistically, putting out other fires =P

Nonetheless, some points I like to add in PR descriptions: - why is there a problem with the current approach? - is this the minimal viable change, or is it trying to change too many things at once? The former is way better, in general. - what are the trade-offs? This PR is using more memory to lower the runtime, but I hadn't measure it yet when I opened it. - Not changing public APIs is always good to convince reviewers. If the project follows a semantic versioning scheme, changes to the public APIs are major version bumps, and that can brings other consequences for users.

Setting up for changing code

If this was a bug fix PR, the first thing I would do is write a new test triggering the bug, and then proceed to fix it in the code (Hmm, maybe that would be another good walkthrough?). But this PR is making performance claims ("it's going to be faster"), and that's a bit hard to codify in tests. ¹ Since it's also proposing to change a method (_signatures in LCA indices) that is better to benchmark with a real index (and not a toy example), I used the same data and command I run in sourmash_resources to check how memory consumption and runtime changed. For reference, this is the command:

sourmash search -o out.csv --scaled 2000 -k 51 HSMA33OT.fastq.gz.sig genbank-k51.lca.json.gz

I'm using the benchmark feature from snakemake in sourmash_resources to track how much memory, runtime and I/O is used for each command (and version) of sourmash, and generate the plots in the README in that repo. That is fine for a high-level view ("what's the maximum memory used?"), but not so useful for digging into details ("what method is consuming most memory?").

Another additional problem is the dual² language nature of sourmash, where we have Python calling into Rust code (via CFFI). There are great tools for measuring and profiling Python code, but they tend to not work with extension code...

So, let's bring two of my favorite tools to help!

Memory profiling: heaptrack

heaptrack is a heap profiler, and I first heard about it from Vincent Prouillet. It's main advantage over other solutions (like valgrind's massif) is the low overhead and... how easy it is to use: just stick heaptrack in front of your command, and you're good to go!

Example output:

$ heaptrack sourmash search -o out.csv --scaled 2000 -k 51 HSMA33OT.fastq.gz.sig genbank-k51.lca.json.gz

heaptrack stats:
        allocations:            1379353
        leaked allocations:     1660
        temporary allocations:  168984
Heaptrack finished! Now run the following to investigate the data:

  heaptrack --analyze heaptrack.sourmash.66565.gz

heaptrack --analyze is a very nice graphical interface for analyzing the results, but for this PR I'm mostly focusing on the Summary page (and overall memory consumption). Tracking allocations in Python doesn't give many details, because it shows the CPython functions being called, but the ability to track into the extension code (Rust) allocations is amazing for finding bottlenecks (and memory leaks =P). ³

CPU profiling: py-spy

Just as other solutions exist for profiling memory, there are many for profiling CPU usage in Python, including profile and cProfile in the standard library. Again, the issue is being able to analyze extension code, and bringing the cannon (the perf command in Linux, for example) looses the benefit of tracking Python code properly (because we get back the CPython functions, not what you defined in your Python code).

Enters py-spy by Ben Frederickson, based on the rbspy project by Julia Evans. Both use a great idea: read the process maps for the interpreters and resolve the full stack trace information, with low overhead (because it uses sampling). py-spy also goes further and resolves [native Python extensions] stack traces, meaning we can get the complete picture all the way from the Python CLI to the Rust core library!⁴

py-spy is also easy to use: stick py-spy record --output search.svg -n -- in front of the command, and it will generate a flamegraph in search.svg. The full command for this PR is

py-spy record --output search.svg -n -- sourmash search -o out.csv --scaled 2000 -k 51 HSMA.fastq.sig genbank-k51.lca.json.gz

Show me the code!

OK, OK, sheesh. But it's worth repeating: the code is important, but there are many other aspects that are just as important =]

Replacing add_hash calls with one add_many

Let's start at the _signatures() method on LCA indices. This is the original method:

@cached_property
def _signatures(self):
    "Create a _signatures member dictionary that contains {idx: minhash}."
    from .. import MinHash

    minhash = MinHash(n=0, ksize=self.ksize, scaled=self.scaled)

    debug('creating signatures for LCA DB...')
    sigd = defaultdict(minhash.copy_and_clear)

    for (k, v) in self.hashval_to_idx.items():
        for vv in v:
            sigd[vv].add_hash(k)

    debug('=> {} signatures!', len(sigd))
    return sigd

sigd[vv].add_hash(k) is the culprit. Each call to .add_hash has to go thru CFFI to reach the extension code, and the overhead is significant. It's similar situation to accessing array elements in NumPy: it works, but it is way slower than using operations that avoid crossing from Python to the extension code. What we want to do instead is call .add_many(hashes), which takes a list of hashes and process it entirely in Rust (ideally. We will get there).

But, to have a list of hashes, there is another issue with this code.

for (k, v) in self.hashval_to_idx.items():
    for vv in v:
        sigd[vv].add_hash(k)

There are two nested for loops, and add_hash is being called with values from the inner loop. So... we don't have the list of hashes beforehand.

But we can change the code a bit to save the hashes for each signature in a temporary list, and then call add_many on the temporary list. Like this:

temp_vals = defaultdict(list)

for (k, v) in self.hashval_to_idx.items():
    for vv in v:
        temp_vals[vv].append(k)

for sig, vals in temp_vals.items():
    sigd[sig].add_many(vals)

There is a trade-off here: if we save the hashes in temporary lists, will the memory consumption be so high that the runtime gains of calling add_many in these temporary lists be cancelled?

Time to measure it =]

Wait, it got worse?!?! Building temporary lists only takes time and memory, and bring no benefits!

This mystery goes away when you look at the add_many method:

def add_many(self, hashes):
    "Add many hashes in at once."
    if isinstance(hashes, MinHash):
        self._methodcall(lib.kmerminhash_add_from, hashes._objptr)
    else:
        for hash in hashes:
            self._methodcall(lib.kmerminhash_add_hash, hash)

The first check in the if statement is a shortcut for adding hashes from another MinHash, so let's focus on else part... And turns out that add_many is lying! It doesn't process the hashes in the Rust extension, but just loops and call add_hash for each hash in the list. That's not going to be any faster than what we were doing in _signatures.

Time to fix add_many!

Oxidizing add_many

The idea is to change this loop in add_many:

for hash in hashes:
    self._methodcall(lib.kmerminhash_add_hash, hash)

with a call to a Rust extension function:

self._methodcall(lib.kmerminhash_add_many, list(hashes), len(hashes))

self._methodcall is a convenience method defined in [RustObject] which translates a method-like call into a function call, since our C layer only has functions. This is the C prototype for this function:

void kmerminhash_add_many(
    KmerMinHash *ptr,
    const uint64_t *hashes_ptr,
    uintptr_t insize
  );

You can almost read it as a Python method declaration, where KmerMinHash *ptr means the same as the self in Python methods. The other two arguments are a common idiom when passing pointers to data in C, with insize being how many elements we have in the list. ⁵. CFFI is very good at converting Python lists into pointers of a specific type, as long as the type is of a primitive type (uint64_t in our case, since each hash is a 64-bit unsigned integer number).

And the Rust code with the implementation of the function:

ffi_fn! {
unsafe fn kmerminhash_add_many(
    ptr: *mut KmerMinHash,
    hashes_ptr: *const u64,
    insize: usize,
  ) -> Result<()> {
    let mh = {
        assert!(!ptr.is_null());
        &mut *ptr
    };

    let hashes = {
        assert!(!hashes_ptr.is_null());
        slice::from_raw_parts(hashes_ptr as *mut u64, insize)
    };

    for hash in hashes {
      mh.add_hash(*hash);
    }

    Ok(())
}
}

Let's break what's happening here into smaller pieces. Starting with the function signature:

ffi_fn! {
unsafe fn kmerminhash_add_many(
    ptr: *mut KmerMinHash,
    hashes_ptr: *const u64,
    insize: usize,
  ) -> Result<()>

The weird ffi_fn! {} syntax around the function is a macro in Rust: it changes the final generated code to convert the return value (Result<()>) into something that is valid C code (in this case, void). What happens if there is an error, then? The Rust extension has code for passing back an error code and message to Python, as well as capturing panics (when things go horrible bad and the program can't recover) in a way that Python can then deal with (raising exceptions and cleaning up). It also sets the #[no_mangle] attribute in the function, meaning that the final name of the function will follow C semantics (instead of Rust semantics), and can be called more easily from C and other languages. This ffi_fn! macro comes from symbolic, a big influence on the design of the Python/Rust bridge in sourmash.

unsafe is the keyword in Rust to disable some checks in the code to allow potentially dangerous things (like dereferencing a pointer), and it is required to interact with C code. unsafe doesn't mean that the code is always unsafe to use: it's up to whoever is calling this to verify that valid data is being passed and invariants are being preserved.

If we remove the ffi_fn! macro and the unsafe keyword, we have

fn kmerminhash_add_many(
    ptr: *mut KmerMinHash,
    hashes_ptr: *const u64,
    insize: usize
  );

At this point we can pretty much map between Rust and the C function prototype:

void kmerminhash_add_many(
    KmerMinHash *ptr,
    const uint64_t *hashes_ptr,
    uintptr_t insize
  );

Some interesting points:

• We use fn to declare a function in Rust.
• The type of an argument comes after the name of the argument in Rust, while it's the other way around in C. Same for the return type (it is omitted in the Rust function, which means it is -> (), equivalent to a void return type in C).
• In Rust everything is immutable by default, so we need to say that we want a mutable pointer to a KmerMinHash item: *mut KmerMinHash). In C everything is mutable by default.
• u64 in Rust -> uint64_t in C
• usize in Rust -> uintptr_t in C

Let's check the implementation of the function now. We start by converting the ptr argument (a raw pointer to a KmerMinHash struct) into a regular Rust struct:

let mh = {
    assert!(!ptr.is_null());
    &mut *ptr
};

This block is asserting that ptr is not a null pointer, and if so it dereferences it and store in a mutable reference. If it was a null pointer the assert! would panic (which might sound extreme, but is way better than continue running because dereferencing a null pointer is BAD). Note that functions always need all the types in arguments and return values, but for variables in the body of the function Rust can figure out types most of the time, so no need to specify them.

The next block prepares our list of hashes for use:

let hashes = {
    assert!(!hashes_ptr.is_null());
    slice::from_raw_parts(hashes_ptr as *mut u64, insize)
};

We are again asserting that the hashes_ptr is not a null pointer, but instead of dereferencing the pointer like before we use it to create a slice, a dynamically-sized view into a contiguous sequence. The list we got from Python is a contiguous sequence of size insize, and the slice::from_raw_parts function creates a slice from a pointer to data and a size.

Oh, and can you spot the bug? I created the slice using *mut u64, but the data is declared as *const u64. Because we are in an unsafe block Rust let me change the mutability, but I shouldn't be doing that, since we don't need to mutate the slice. Oops.

Finally, let's add hashes to our MinHash! We need a for loop, and call add_hash for each hash:

for hash in hashes {
  mh.add_hash(*hash);
}

Ok(())

We finish the function with Ok(()) to indicate no errors occurred.

Why is calling add_hash here faster than what we were doing before in Python? Rust can optimize these calls and generate very efficient native code, while Python is an interpreted language and most of the time don't have the same guarantees that Rust can leverage to generate the code. And, again, calling add_hash here doesn't need to cross FFI boundaries or, in fact, do any dynamic evaluation during runtime, because it is all statically analyzed during compilation.

Putting it all together

And... that's the PR code. There are some other unrelated changes that should have been in new PRs, but since they were so small it would be more work than necessary. OK, that's a lame excuse: it's confusing for reviewers to see these changes here, so avoid doing that if possible!

But, did it work?

We are using 200 MB of extra memory, but taking less than half the time it was taking before. I think this is a good trade-off, and so did the reviewer and the PR was approved.

Hopefully this was useful, 'til next time!

Comments?

• Thread on Mastodon
• Thread on Twitter
• Lobste.rs submission

Bonus: list or set?

The first version of the PR used a set instead of a list to accumulate hashes. Since a set doesn't have repeated elements, this could potentially use less memory. The code:

temp_vals = defaultdict(set)

for (k, v) in self.hashval_to_idx.items():
    for vv in v:
        temp_vals[vv].add(k)

for sig, vals in temp_vals.items():
    sigd[sig].add_many(vals)

The runtime was again half of the original, but... | version | mem | time | | :-- | :-- | :-- | |original|1.5 GB|160s| |set|3.8GB|80s| |list|1.7GB|73s| ... memory consumption was almost 2.5 times the original! WAT

The culprit this time? The new kmerminhash_add_many call in the add_many method. This one:

self._methodcall(lib.kmerminhash_add_many, list(hashes), len(hashes))

CFFI doesn't know how to convert a set into something that C understands, so we need to call list(hashes) to convert it into a list. Since Python (and CFFI) can't know if the data is going to be used later ⁶ it needs to keep it around (and be eventually deallocated by the garbage collector). And that's how we get at least double the memory being allocated...

There is another lesson here. If we look at the for loop again:

for (k, v) in self.hashval_to_idx.items():
    for vv in v:
        temp_vals[vv].add(k)

each k is already unique because they are keys in the hashval_to_idx dictionary, so the initial assumption (that a set might save memory because it doesn't have repeated elements) is... irrelevant for the problem =]

Footnotes

1. We do have https://asv.readthedocs.io/ set up for micro-benchmarks, and now that I think about it... I could have started by writing a benchmark for add_many, and then showing that it is faster. I will add this approach to the sourmash PR checklist =] ↩

2. or triple, if you count C ↩

3. It would be super cool to have the unwinding code from py-spy in heaptrack, and be able to see exactly what Python methods/lines of code were calling the Rust parts... ↩

4. Even if py-spy doesn't talk explicitly about Rust, it works very very well, woohoo! ↩

5. Let's not talk about lack of array bounds checks in C... ↩

6. something that the memory ownership model in Rust does, BTW ↩

What is Tsuru?

What is Tsuru public cloud? What does "beta availability" means?

Which development platforms are going to be available?

How limited is it going to be?

When will it be available?

What if I don't want to wait?

% curl -O http://nginx.org/download/nginx-1.0.9.tar.gz
% tar -xzf nginx-1.0.9.tar.gz

% ./configure --prefix=/opt/nginx --user=nginx --group=nginx
% make
% [sudo] make install

% [sudo] adduser --system --no-create-home --disabled-login --disabled-password --group nginx

% [sudo] /opt/nginx/sbin/nginx

Linode provides an init script that uses start-stop-daemon, you might want to use it.

nginx configuration

• nginx should start workers with the nginx user
• nginx should have two worker processes
• the PID should be stored in the /opt/nginx/log/nginx.pid file
• nginx must have an access log in /opt/nginx/logs/access.log
• the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the nginx.conf file (assume that the library project is in the directory /opt/projects).

user  nginx;
worker_processes  2;

pid logs/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile           on;
    keepalive_timeout  65;

    include /opt/projects/showcase/nginx.conf;
}

server {
    listen 80;
    server_name localhost;

    charset utf-8;

    location / {
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    Host        $http_host;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_pass http://localhost:8000;
    }

    location /static {
        root /opt/projects/showcase/;
        expires 1d;
    }
}

• X-Real-IP: forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)
• Host: the Host header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints
• X-Forwarded-For: also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.

Python
======

--------------

If
==

* Please don't use ()
* Never forget the ``:`` at the end of the line

Check this code:

.. sourcecode:: python

    x, y = 1, 2
    if x > y:
        print 'x is greater'

--------------

For
===

* ``for`` iterates over a sequence
* Never forget the ``:`` at the end of the line

Check this code:

.. sourcecode:: python

    numbers = [1, 2, 3, 4, 5,]
    for number in numbers:
        print number

--------------

While
=====

* ``while`` is like ``if``, but executes while the codition is ``True``
* please don't use ()
* never forget the ``:`` at the end of the line

Check this code:

.. sourcecode:: python

    from random import randint

    args = (1, 10,)
    x = randint(*args)
    while x != 6:
        x = randint(*args)

--------------

Thank you!
==========

% landslide python.rst

% [sudo] pip install landslide

% landslide --help

What is a splinter sprint?

How to contribute

1. Choose an issue
2. Create a fork
3. Send a pull request

Preparing for the sprint

.
├── SpecRunner.html
├── lib
│   ├── jasmine-1.0.2
│   │   ├── MIT.LICENSE
│   │   ├── jasmine-html.js
│   │   ├── jasmine.css
│   │   └── jasmine.js
│   └── jquery-1.6.1.min.js
├── spec
│   └── ZebrafySpec.js
└── src
    └── jquery.zebrafy.js

describe('Zebrafy', function () {

});

it('should apply classes zebrafy-odd and zebrafy-even to each other table lines', function () {
    var table = $("#zebra-table");
    table.zebrafy();
    expect(table).toBeZebrafyied();
});

beforeEach(function () {
    $('<table id="zebra-table"></table>').appendTo('body');
    for (var i=0; i < 10; i++) {
        $('<tr></tr>').append('<td></td>').append('<td></td>').append('<td></td>').appendTo('#zebra-table');
    };
});

afterEach(function () {
    $("#zebra-table").remove();
});

it('should apply classes zebrafy-odd and zebrafy-even to each other table lines', function () {
    var table = $("#zebra-table");
    table.zebrafy();
    expect(table).toBeZebrafyied();
});

beforeEach(function () {
    this.addMatchers({
        toBeZebrafyied: function() {
            var isZebrafyied = true;

            this.actual.find("tr:even").each(function (index, tr) {
                isZebrafyied = $(tr).hasClass('zebrafy-odd') === false && $(tr).hasClass('zebrafy-even');
                if (!isZebrafyied) {
                    return;
                };
            });

            this.actual.find("tr:odd").each(function (index, tr) {
                isZebrafyied = $(tr).hasClass('zebrafy-odd') && $(tr).hasClass('zebrafy-even') === false;
                if (!isZebrafyied) {
                    return;
                };
            });

            return isZebrafyied;
        }
    });
});

(function ($) {
    $.fn.zebrafy = function () {
        this.find("tr:even").addClass("zebrafy-even");
        this.find("tr:odd").addClass("zebrafy-odd");
    };
})(jQuery);

it('zebrafy should be chainable', function() {
    var table = $("#zebra-table");
    table.zebrafy().addClass('black-bg');
    expect(table.hasClass('black-bg')).toBeTruthy();
});

(function ($) {
    $.fn.zebrafy = function () {
        return this.each(function (index, table) {
            $(table).find("tr:even").addClass("zebrafy-even");
            $(table).find("tr:odd").addClass("zebrafy-odd");
        });
    };
})(jQuery);

browser = Browser("firefox")

browser.visit("https://www.facebook.com")

def do_login_if_need(self, username, password):
    if self.browser.is_element_present_by_css('div.menu_login_container'):
        self.browser.fill('email', username)
        self.browser.fill('pass', password)
        self.browser.find_by_css('div.menu_login_container input[type="submit"]').first.click()
        assert self.browser.is_element_present_by_css('li#navAccount')

1. On Facebook homepage, click on “Events” link, of left menu
2. The “Events” page will load, so click on “Create an Event” button
3. The user see a page with a form to create an event
4. Fill the date and chose the time
5. Define what is the name of the event, where it will happen and write a short description for it
6. Invite some guests
7. Upload a picture for the event
8. Click on “Create Event” button

browser.find_by_css('li#navItem_events a').first.click()

browser.fill('event_startIntlDisplay', '5/21/2011')
browser.select('start_time_min', '480')
browser.fill('name', 'Splinter sprint')
browser.fill('location', 'Rio de Janeiro, Brazil')
browser.fill('desc', 'For more info, check out the #cobratem channel on freenode!')

<select name="gender">
    <option value="m">Male</option>
    <option value="f">Female</option>
</select>

browser.select("gender", "m")

browser.find_by_css('div.eventEditUpload a.uiButton').first.click()

if not browser.is_element_present_by_css('iframe#upload_pic_frame', wait_time=10):
fail("The upload pic iframe did'n't appear :(")

Important: fail is a pseudocode sample and doesn’t exist (if you’re using unittest library, you can invoke self.fail in a TestCase, exactly what I did in complete snippet for this example, available at Github).

with browser.get_iframe('upload_pic_frame') as frame:
    frame.attach_file('pic', picture_path)
    time.sleep(10)

browser.find_by_css('label.uiButton input[type="submit"]').first.click()
title = browser.find_by_css('h1 span').first.text
assert title == 'Splinter sprint'

nginx

% ./configure --prefix=/opt/nginx --user=nginx --group=nginx --with-http_ssl_module --add-module={your memc_module source path}
% make
% sudo make install

user  nginx;
worker_processes  1;

error_log  logs/error.log;

events {
    worker_connections  1024;
}

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                                '$status $body_bytes_sent "$http_referer" '
                                '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include /opt/nginx/sites-enabled/*;
}

server {
    listen       80;
    server_name  localhost;

    default_type  text/html;

    location / {
        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    Host        $http_host;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;

        if ($request_method = POST) {
            proxy_pass      http://localhost:8080;
            break;
        }

        set $memcached_key   "$uri";
        memcached_pass      127.0.0.1:11211;

        error_page  501 404 502 = /fallback$uri;
    }

    location /fallback/ {
        internal;    

        proxy_set_header    X-Real-IP   $remote_addr;
        proxy_set_header    Host        $http_host;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_redirect      off;

        proxy_pass          http://localhost:8080;
    }

}

• 404: memcached module returns a 404 error when the key is not on memcached server;
• 502: memcached module returns a 502 error when it can’t found memcached server.

Java application

package com.franciscosouza.memcached.filter;

import java.io.IOException;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.net.InetSocketAddress;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;

import net.spy.memcached.MemcachedClient;

/**
 * Servlet Filter implementation class MemcachedFilter
 */
public class MemcachedFilter implements Filter {

    private MemcachedClient mmc;

    static class MemcachedHttpServletResponseWrapper extends HttpServletResponseWrapper {

        private StringWriter sw = new StringWriter();

        public MemcachedHttpServletResponseWrapper(HttpServletResponse response) {
            super(response);
        }

        public PrintWriter getWriter() throws IOException {
            return new PrintWriter(sw);
        }

        public ServletOutputStream getOutputStream() throws IOException {
            throw new UnsupportedOperationException();
        }

        public String toString() {
            return sw.toString();
        }
    }

    /**
     * Default constructor.
     */
    public MemcachedFilter() {
    }

    /**
     * @see Filter#destroy()
     */
    public void destroy() {
    }

    /**
     * @see Filter#doFilter(ServletRequest, ServletResponse, FilterChain)
     */
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {
        MemcachedHttpServletResponseWrapper wrapper = new MemcachedHttpServletResponseWrapper((HttpServletResponse) response);
        chain.doFilter(request, wrapper);

        HttpServletRequest inRequest = (HttpServletRequest) request;
        HttpServletResponse inResponse = (HttpServletResponse) response;

        String content = wrapper.toString();

        PrintWriter out = inResponse.getWriter();
        out.print(content);

        if (!inRequest.getMethod().equals("POST")) {
            String key = inRequest.getRequestURI();
            mmc.set(key, 5, content);
        }
    }

    /**
     * @see Filter#init(FilterConfig)
     */
    public void init(FilterConfig fConfig) throws ServletException {
        try {
            mmc = new MemcachedClient(new InetSocketAddress("localhost", 11211));
        } catch (IOException e) {
            e.printStackTrace();
            throw new ServletException(e);
        }
    }
}

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
    <display-name>memcached sample</display-name>

    <filter>
        <filter-name>vraptor</filter-name>
        <filter-class>br.com.caelum.vraptor.VRaptor</filter-class>
    </filter>
    
    <filter>
        <filter-name>memcached</filter-name>
        <filter-class>com.franciscosouza.memcached.filter.MemcachedFilter</filter-class>
    </filter>
    
    <filter-mapping>
        <filter-name>memcached</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>

    <filter-mapping>
        <filter-name>vraptor</filter-name>
        <url-pattern>/*</url-pattern>
        <dispatcher>FORWARD</dispatcher>
        <dispatcher>REQUEST</dispatcher>
    </filter-mapping>

</web-app>

% wget http://www.tipfy.org/tipfy.build.tar.gz
% tar -xvzf tipfy.0.6.2.build.tar.gz
% mv project gaeseries

application: gaeseries
version: 4
runtime: python
api_version: 1

derived_file_type:
- python_precompiled

handlers:
- url: /(robots\.txt|favicon\.ico)
  static_files: static/\1
  upload: static/(.*)

- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

- url: /_ah/queue/deferred
  script: main.py
  login: admin

- url: /.* 
 script: main.py

% mkdir blog
% touch blog/__init__.py

# -*- coding: utf-8 -*-
"""
    config
    ~~~~~~

    Configuration settings.

    :copyright: 2009 by tipfy.org.
    :license: BSD, see LICENSE for more details.
"""
config = {}

# Configurations for the 'tipfy' module.
config['tipfy'] = {
    # Enable debugger. It will be loaded only in development.
    'middleware': [
        'tipfy.ext.debugger.DebuggerMiddleware',
    ],
    # Enable the Hello, World! app example.
    'apps_installed': [
        'apps.hello_world',
        'apps.blog',
    ],
}

from google.appengine.ext import db

class Post(db.Model):
    title = db.StringProperty(required = True)
    content = db.TextProperty(required = True)
    when = db.DateTimeProperty(auto_now_add = True)
    author = db.UserProperty(required = True)

# -*- coding: utf-8 -*-
from tipfy import RequestHandler
from tipfy.ext.jinja2 import render_response
from models import Post

class PostListingHandler(RequestHandler):
    def get(self):
        posts = Post.all()
        return render_response('list_posts.html', posts=posts)

<html>
    <head>
      <meta http-equiv="Content-type" content="text/html; charset=utf-8"/>
      <title>{% block title %}{% endblock %}</title>
    </head>
    <body id="">
        {% block content %}{% endblock %}
    </body>
</html>

{% extends "base.html" %}

{% block title %}
    Posts list
{% endblock %}

{% block content %}
    Listing all posts:

    <ul>
        {% for post in posts %}
            <li>
                {{ post.title }} (written by {{ post.author.nickname() }})
                {{ post.content }}
            </li>
        {% endfor %}
    </ul>
{% endblock %}

from tipfy import Rule

def get_rules(app):
    rules = [
        Rule('/posts', endpoint='post-listing', handler='apps.blog.handlers.PostListingHandler'),
    ]

    return rules

% /usr/local/google_appengine/dev_appserver.py app

config['tipfy.ext.session'] = {
    'secret_key' : 'just_dev_testH978DAGV9B9sha_W92S',
}

% wget http://bitbucket.org/simplecodes/wtforms/get/tip.tar.bz2
% tar -xvf tip.tar.bz2
% cp -r wtforms/wtforms/ ~/Projetos/gaeseries/app/lib/
% wget http://pypi.python.org/packages/source/t/tipfy.ext.wtforms/tipfy.ext.wtforms-0.6.tar.gz
% tar -xvzf tipfy.ext.wtforms-0.6.tar.gz
% cp -r tipfy.ext.wtforms-0.6/tipfy ~/Projetos/gaeseries/app/distlib

class PostForm(Form):
    csrf_protection = True
    title = fields.TextField('Title', validators=[validators.Required()])
    content = fields.TextAreaField('Content', validators=[validators.Required()])

class NewPostHandler(RequestHandler, AppEngineAuthMixin, AllSessionMixins):
    middleware = [SessionMiddleware]

    @login_required
    def get(self, **kwargs):
        return render_response('new_post.html', form=self.form)

    @login_required
    def post(self, **kwargs):
        if self.form.validate():
            post = Post(
                title = self.form.title.data,
                content = self.form.content.data,
                author = self.auth_session
            )
            post.put()
            return redirect('/posts')
        return self.get(**kwargs)

    @cached_property
    def form(self):
        return PostForm(self.request)

{% extends "base.html" %}

{% block title %}
    New post
{% endblock %}

{% block content %}
    <form action="" method="post" accept-charset="utf-8">
        <p>
            <label for="title">{{ form.title.label }}</label>

            {{ form.title|safe }}

            {% if form.title.errors %}
            <ul class="errors">
                {% for error in form.title.errors %}
                <li>{{ error }}</li>
                {% endfor %}
            </ul>
            {% endif %}
        </p>
        <p>
            <label for="content">{{ form.content.label }}</label>

            {{ form.content|safe }}

            {% if form.content.errors %}
            <ul class="errors">
                {% for error in form.content.errors %}
                <li>{{ error }}</li>
                {% endfor %}
            </ul>
            {% endif %}
        </p>
        <p><input type="submit" value="Save post"/></p>
    </form>
{% endblock %}

% /usr/local/google_appengine/appcfg.py update app

application: gaeseries
version: 3
runtime: python
api_version: 1

handlers:
- url: .*
  script: main.py

% wget http://github.com/mitsuhiko/flask/zipball/0.6
% unzip mitsuhiko-flask-0.6-0-g5cadd9d.zip
% cp -r mitsuhiko-flask-5cadd9d/flask ~/Projetos/blog/gaeseries
% wget http://pypi.python.org/packages/source/W/Werkzeug/Werkzeug-0.6.2.tar.gz
% tar -xvzf Werkzeug-0.6.2.tar.gz
% cp -r Werkzeug-0.6.2/werkzeug ~/Projetos/blog/gaeseries/
% wget http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.tar.gz
% tar -xvzf Jinja2-2.5.tar.gz
% cp -r Jinja2-2.5/jinja2 ~/Projetos/blog/gaeseries/
% wget http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.1.tar.gz
% tar -xvzf simplejson-2.1.1.tar.gz
% cp -r simplejson-2.1.1/simplejson ~/Projetos/blog/gaeseries/

% mkdir blog
% touch blog/__init__.py

from flask import Flask
import settings

app = Flask('blog')
app.config.from_object('blog.settings')

import views

% touch blog/views.py
% touch blog/settings.py

DEBUG=True
SECRET_KEY='dev_key_h8hfne89vm'
CSRF_ENABLED=True
CSRF_SESSION_LKEY='dev_key_h8asSNJ9s9=+'

from google.appengine.ext import db

class Post(db.Model):
    title = db.StringProperty(required = True)
    content = db.TextProperty(required = True)
    when = db.DateTimeProperty(auto_now_add = True)
    author = db.UserProperty(required = True)

from blog import app
from models import Post
from flask import render_template

@app.route('/posts')
def list_posts():
    posts = Post.all()
    return render_template('list_posts.html', posts=posts)

<html>
    <head>
        <meta http-equiv="Content-type" content="text/html; charset=utf-8"/>
        <title>{% block title %}Blog{% endblock %}</title>
    </head>
    <body>
        {% block content %}{% endblock %}
    </body>
</html>

{% extends "base.html" %}

{% block content %}
<ul>
    {% for post in posts %}
    <li>
        {{ post.title }} (written by {{ post.author.nickname() }})

        {{ post.content }}
    </li>
    {% endfor %}
</ul>
{% endblock %}

from google.appengine.ext.webapp.util import run_wsgi_app
from blog import app

run_wsgi_app(app)

% /usr/local/google_appengine/dev_appserver.py .

from functools import wraps
from google.appengine.api import users
from flask import redirect, request

def login_required(func):
    @wraps(func)
    def decorated_view(*args, **kwargs):
        if not users.get_current_user():
            return redirect(users.create_login_url(request.url))
        return func(*args, **kwargs)
    return decorated_view

% wget http://pypi.python.org/packages/source/W/WTForms/WTForms-0.6.zip
% unzip WTForms-0.6.zip
% cp -r WTForms-0.6/wtforms ~/Projetos/blog/gaeseries/
% wget http://pypi.python.org/packages/source/F/Flask-WTF/Flask-WTF-0.2.3.tar.gz
% tar -xvzf Flask-WTF-0.2.3.tar.gz
% cp -r Flask-WTF-0.2.3/flaskext ~/Projetos/blog/gaeseries/

from flaskext import wtf
from flaskext.wtf import validators

class PostForm(wtf.Form):
    title = wtf.TextField('Title', validators=[validators.Required()])
    content = wtf.TextAreaField('Content', validators=[validators.Required()])

@app.route('/posts/new', methods = ['GET', 'POST'])
@login_required
def new_post():
    form = PostForm()
    if form.validate_on_submit():
        post = Post(title = form.title.data,
                    content = form.content.data,
                    author = users.get_current_user())
        post.put()
        flash('Post saved on database.')
        return redirect(url_for('list_posts'))
    return render_template('new_post.html', form=form)

{% extends "base.html" %}

{% block content %}
    <h1 id="">Write a post</h1>
    <form action="{{ url_for('new_post') }}" method="post" accept-charset="utf-8">
        {{ form.csrf_token }}
        <p>
            <label for="title">{{ form.title.label }}</label>

            {{ form.title|safe }}

            {% if form.title.errors %}
            <ul class="errors">
                {% for error in form.title.errors %}
                <li>{{ error }}</li>
                {% endfor %}
            </ul>
            {% endif %}
        </p>
        <p>
            <label for="content">{{ form.content.label }}</label>

            {{ form.content|safe }}

            {% if form.content.errors %}
            <ul class="errors">
                {% for error in form.content.errors %}
                <li>{{ error }}</li>
                {% endfor %}
            </ul>
            {% endif %}
        </p>
        <p><input type="submit" value="Save post"/></p>
    </form>
{% endblock %}

% /usr/local/google_appengine/appcfg.py update .

application: gaeseries
version: 2
api_version: 1
runtime: python

handlers:

- url: /(?P<a>.+?)/static/(?P<b>.+)
  static_files: applications/\1/static/\2
  upload: applications/(.+?)/static/(.+)
  secure: optional
  expiration: "90d"

- url: /admin-gae/.*
  script: $PYTHON_LIB/google/appengine/ext/admin
  login: admin
 
- url: /_ah/queue/default
  script: gaehandler.py
  login: admin

- url: .*
  script: gaehandler.py  
  secure: optional

skip_files: |
^(.*/)?(
 (app\.yaml)|
 (app\.yml)|
 (index\.yaml)|
 (index\.yml)|
 (#.*#)|
 (.*~)|
 (.*\.py[co])|
 (.*/RCS/.*)|
 (\..*)|
 ((admin|examples|welcome)\.tar)|
 (applications/(admin|examples)/.*)|
 (applications/.*?/databases/.*) |
 (applications/.*?/errors/.*)|
 (applications/.*?/cache/.*)|
 (applications/.*?/sessions/.*)|
 )$

current_user_id = (auth.user and auth.user.id) or 0

db.define_table('posts', db.Field('title'),
                    db.Field('content', 'text'),
                    db.Field('author', db.auth_user, default=current_user_id, writable=False),
                    db.Field('date', 'datetime', default=request.now, writable=False)
                )

db.posts.title.requires = IS_NOT_EMPTY()
db.posts.content.requires = IS_NOT_EMPTY()

def index():
    posts = db().select(db.posts.ALL)
    return response.render('posts/index.html', locals())

{{extend 'layout.html'}}
<h1 id="">Listing all posts</h1>
<dl>
    {{for post in posts:}}
    <dt>{{=post.title}} (written by {{=post.author.first_name}})</dt>
    <dd>{{=post.content}}</dd>
    {{pass}}
</dl>

% /usr/local/google_appengine/dev_appserver.py .

@auth.requires_login()
def new():
    form = SQLFORM(db.posts, fields=['title','content'])
    if form.accepts(request.vars, session):
        response.flash = 'Post saved.'
        redirect(URL('blog', 'posts', 'index'))
    return response.render('posts/new.html', dict(form=form))

{{extend 'layout.html'}}

<h1 id="">
Save a new post</h1>
{{=form}}

% /usr/local/google_appengine/appcfg.py update .

% juju bootstrap
% juju deploy mysql
% juju deploy wordpress
% juju add-relation wordpress mysql
% juju expose wordpress

import subprocess
import traceback

from cloudinit import CloudConfig, util

frequency = CloudConfig.per_instance


def yum_install(packages):
    cmd = ["yum", "--quiet", "--assumeyes", "install"]
    cmd.extend(packages)
    subprocess.check_call(cmd)


def handle(_name, cfg, _cloud, log, args):
    pkglist = util.get_cfg_option_list_or_str(cfg, "packages", [])

    if pkglist:
        try:
            yum_install(pkglist)
        except subprocess.CalledProcessError:
            log.warn("Failed to install yum packages: %s" % pkglist)
            log.debug(traceback.format_exc())
            raise

    return True

#cloud-config
modules:
 - yum_packages
packages: [emacs-nox]

--- cloudinit/util.py 2012-05-22 12:18:21.000000000 -0300
+++ cloudinit/util.py 2012-05-31 12:44:24.000000000 -0300
@@ -227,7 +227,7 @@
         stderr=subprocess.PIPE, stdin=subprocess.PIPE)
     out, err = sp.communicate(input_)
     if sp.returncode is not 0:
-        raise subprocess.CalledProcessError(sp.returncode, args, (out, err))
+        raise subprocess.CalledProcessError(sp.returncode, args)
     return(out, err)

diff -u juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py juju-0.5-bzr531/juju/providers/common/cloudinit.py
--- juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py 2012-05-31 15:42:17.480769486 -0300
+++ juju-0.5-bzr531/juju/providers/common/cloudinit.py 2012-05-31 15:55:13.342884919 -0300
@@ -324,3 +324,32 @@
             "machine-id": self._machine_id,
             "juju-provider-type": self._provider_type,
             "juju-zookeeper-hosts": self._join_zookeeper_hosts()}
+
+
+class CentOSCloudInit(CloudInit):
+
+    def _collect_packages(self):
+        packages = [
+            "bzr", "byobu", "tmux", "python-setuptools", "python-twisted",
+            "python-txaws", "python-zookeeper", "python-devel", "juju"]
+        if self._zookeeper:
+            packages.extend([
+                "zookeeper", "libzookeeper", "libzookeeper-devel"])
+        return packages
+
+    def render(self):
+        """Get content for a cloud-init file with appropriate specifications.
+
+        :rtype: str
+
+        :raises: :exc:`juju.errors.CloudInitError` if there isn't enough
+            information to create a useful cloud-init.
+        """
+        self._validate()
+        return format_cloud_init(
+            self._ssh_keys,
+            packages=self._collect_packages(),
+            repositories=self._collect_repositories(),
+            scripts=self._collect_scripts(),
+            data=self._collect_machine_data(),
+            modules=["ssh", "yum_packages", "runcmd"])

diff -ur juju-0.5-bzr531.orig/juju/providers/common/utils.py juju-0.5-bzr531/juju/providers/common/utils.py
--- juju-0.5-bzr531.orig/juju/providers/common/utils.py 2012-05-31 15:42:17.480769486 -0300
+++ juju-0.5-bzr531/juju/providers/common/utils.py 2012-05-31 15:44:06.605014021 -0300
@@ -85,7 +85,7 @@
 
 
 def format_cloud_init(
-    authorized_keys, packages=(), repositories=None, scripts=None, data=None):
+    authorized_keys, packages=(), repositories=None, scripts=None, data=None, modules=None):
     """Format a user-data cloud-init file.
 
     This will enable package installation, and ssh access, and script
@@ -117,8 +117,8 @@
         structure.
     """
     cloud_config = {
-        "apt-update": True,
-        "apt-upgrade": True,
+        "apt-update": False,
+        "apt-upgrade": False,
         "ssh_authorized_keys": authorized_keys,
         "packages": [],
         "output": {"all": "| tee -a /var/log/cloud-init-output.log"}}
@@ -136,6 +136,11 @@
     if scripts:
         cloud_config["runcmd"] = scripts
 
+    if modules:
+        cloud_config["modules"] = modules
+
     output = safe_dump(cloud_config)
     output = "#cloud-config\n%s" % (output)
     return output

• Juju needs a user called ubuntu to interact with its machines, so you will need to create this user in your CentOS AMI/template.
• You need to host all RPM packages for juju, cloud-init and following dependencies in some yum repository (I haven't submitted them to any public repository):
  • python-txaws
  • python-txzookeeper
  • zookeeper
• With this patched Juju, you will have a pure-centos cloud. It does not enable you to have multiple OSes in the same environment.

• Only one philosopher can hold a fork at a time
• It must be impossible for a deadlock to occur
• It must be impossible for a philosopher to starve waiting for a fork
• It must be possible for more than one philosopher to eat at the same time

package main

import (
    "fmt"
    "sync"
    "time"
)

type Fork struct {
    sync.Mutex
}

type Table struct {
    philosophers chan Philosopher
    forks []*Fork
}

func NewTable(forks int) *Table {
    t := new(Table)
    t.philosophers = make(chan Philosopher, forks - 1)
    t.forks = make([]*Fork, forks)
    for i := 0; i < forks; i++ {
        t.forks[i] = new(Fork)
    }
    return t
}

func (t *Table) PushPhilosopher(p Philosopher) {
    p.table = t
    t.philosophers <- data-blogger-escaped-0="" data-blogger-escaped-1="" data-blogger-escaped-2="" data-blogger-escaped-3="" data-blogger-escaped-4="" data-blogger-escaped-:="range" data-blogger-escaped-_="" data-blogger-escaped-able="" data-blogger-escaped-anscombe="" data-blogger-escaped-artin="" data-blogger-escaped-chan="" data-blogger-escaped-e9="" data-blogger-escaped-eat="" data-blogger-escaped-eating...="" data-blogger-escaped-eter="" data-blogger-escaped-f="" data-blogger-escaped-fed.="" data-blogger-escaped-fed="" data-blogger-escaped-fmt.printf="" data-blogger-escaped-for="" data-blogger-escaped-func="" data-blogger-escaped-getforks="" data-blogger-escaped-go="" data-blogger-escaped-heidegger="" data-blogger-escaped-homas="" data-blogger-escaped-index="" data-blogger-escaped-int="" data-blogger-escaped-is="" data-blogger-escaped-leftfork.lock="" data-blogger-escaped-leftfork.unlock="" data-blogger-escaped-leftfork="" data-blogger-escaped-leibniz="" data-blogger-escaped-len="" data-blogger-escaped-lizabeth="" data-blogger-escaped-lombard="" data-blogger-escaped-main="" data-blogger-escaped-make="" data-blogger-escaped-n="" data-blogger-escaped-nagel="" data-blogger-escaped-name="" data-blogger-escaped-ork="" data-blogger-escaped-ottfried="" data-blogger-escaped-p.eat="" data-blogger-escaped-p.fed="" data-blogger-escaped-p.getforks="" data-blogger-escaped-p.name="" data-blogger-escaped-p.putforks="" data-blogger-escaped-p.table.popphilosopher="" data-blogger-escaped-p.table.pushphilosopher="" data-blogger-escaped-p.table="nil" data-blogger-escaped-p.think="" data-blogger-escaped-p="" data-blogger-escaped-philosopher="" data-blogger-escaped-philosopherindex="" data-blogger-escaped-philosophers="" data-blogger-escaped-popphilosopher="" data-blogger-escaped-pre="" data-blogger-escaped-putforks="" data-blogger-escaped-return="" data-blogger-escaped-rightfork.lock="" data-blogger-escaped-rightfork.unlock="" data-blogger-escaped-rightfork="" data-blogger-escaped-s="" data-blogger-escaped-string="" data-blogger-escaped-struct="" data-blogger-escaped-t.forks="" data-blogger-escaped-t="" data-blogger-escaped-table="" data-blogger-escaped-think="" data-blogger-escaped-thinking...="" data-blogger-escaped-time.sleep="" data-blogger-escaped-type="" data-blogger-escaped-was="">
Any feedback is very welcome.

application: gaeseries
version: 1
runtime: python
api_version: 1

default_expiration: '365d'

handlers:
- url: /remote_api
  script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py
  login: admin

- url: /_ah/queue/deferred
  script: djangoappengine/deferred/handler.py
  login: admin

- url: /media/admin
  static_dir: django/contrib/admin/media/

- url: /.*
  script: djangoappengine/main/main.py

% python manage.py startapp core

from django.db import models
from django.contrib.auth.models import User

class Post(models.Model):
    title = models.CharField(max_length = 200)
    content = models.TextField()
    date = models.DateTimeField(auto_now_add = True)
    user = models.ForeignKey(User)

% python manage.py createsuperuser

from django.conf.urls.defaults import *

urlpatterns = patterns('',
    ('^$', 'django.views.generic.simple.direct_to_template',
     {'template': 'home.html'}),

    ('^login/$', 'django.contrib.auth.views.login'),
    ('^logout/$', 'django.contrib.auth.views.logout'),
)

{% extends "base.html" %}

{% block content %}

<p>Fill the form below to login in the system ;)</p>

{% if form.errors %}
<p>Your username and password didn't match. Please try again.</p>
{% endif %}

<form method="post" action="{% url django.contrib.auth.views.login %}">{% csrf_token %}
<table>
<tr>
    <td>{{ form.username.label_tag }}</td>
    <td>{{ form.username }}</td>
</tr>
<tr>
    <td>{{ form.password.label_tag }}</td>
    <td>{{ form.password }}</td>
</tr>
</table>

<input type="submit" value="login" />
<input type="hidden" name="next" value="{{ next }}" />
</form>

{% endblock %}

{% extends "base.html" %}

{% block content %}
    Bye :)
{% endblock %}

LOGIN_URL = '/login/'
LOGOUT_URL = '/logout/'
LOGIN_REDIRECT_URL = '/'

class PostForm(forms.ModelForm):
    class Meta:
        model = Post
        exclude = ('user',)

    def save(self, user, commit = True):
        post = super(PostForm, self).save(commit = False)
        post.user = user

        if commit:
            post.save()

        return post

from django.contrib.auth.decorators import login_required
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponseRedirect
from django.core.urlresolvers import reverse
from forms import PostForm

@login_required
def new_post(request):
    form = PostForm()
    if request.method == 'POST':
        form = PostForm(request.POST)
        if form.is_valid():
            form.save(request.user)
            return HttpResponseRedirect(reverse('core.views.list_posts'))
    return render_to_response('new_post.html',
            locals(), context_instance=RequestContext(request)
    )

def list_posts(request):
    return HttpResponseRedirect('/')

('^posts/new/$', 'core.views.new_post'),
('^posts/$', 'core.views.list_posts'),

{% extends "base.html" %}

{% block content %}
    <form action="{% url core.views.new_post %}" method="post" accept-charset="utf-8">
        {% csrf_token %}
        {{ form.as_p }}
        <p><input type="submit" value="Post!"/></p>
    </form>
{% endblock %}

def list_posts(request):
    posts = Post.objects.all()
    return render_to_response('list_posts.html',
            locals(), context_instance=RequestContext(request)
    )

{% extends "base.html" %}

{% block content %}
<dl>
    {% for post in posts %}
    <dt>{{ post.title }} (written by {{ post.user.username }})</dt>
    <dd>{{ post.content }}</dd>
    {% endfor %}
</dl>
{% endblock %}

% python manage.py deploy

% python manage.py remote createsuperuser

In January I wrote a post for the Rust 2019 call for blogs. The 2020 call is aiming for an RFC and roadmap earlier this time, so here is my 2020 post =]

Last call review: what happened?

An attribute proc-macro like #[wasm_bindgen] but for FFI

This sort of happened... because WebAssembly is growing =]

I was very excited when Interface Types showed up in August, and while it is still very experimental it is moving fast and bringing saner paths for interoperability than raw C FFIs. David Beazley even point this at the end of his PyCon India keynote, talking about how easy is to get information out of a WebAssembly module compared to what had to be done for SWIG.

This doesn't solve the problem where strict C compatibility is required, or for platforms where a WebAssembly runtime is not available, but I think it is a great solution for scientific software (or, at least, for my use cases =]).

"More -sys and Rust-like crates for interoperability with the larger ecosystems" and "More (bioinformatics) tools using Rust!"

I did some of those this year (bbhash-sys and mqf), and also found some great crates to use in my projects. Rust is picking up steam in bioinformatics, being used as the primary choice for high quality software (like varlociraptor, or the many coming from 10X Genomics) but it is still somewhat hard to find more details (I mostly find it on Twitter, and sometime Google Scholar alerts). It would be great to start bringing this info together, which leads to...

"A place to find other scientists?"

Hey, this one happened! Luca Palmieri started a conversation on reddit and the #science-and-ai Discord channel on the Rust community server was born! I think it works pretty well, and Luca also has being doing a great job running workshops and guiding the conversation around rust-ml.

Rust 2021: Interoperability

Rust is amazing because it is very good at bringing many concepts and ideas that seem contradictory at first, but can really shine when synthesized. But can we share this combined wisdom and also improve the situation in other places too? Despite the "Rewrite it in Rust" meme, increased interoperability is something that is already driving a lot of the best aspects of Rust:

• Interoperability with other languages: as I said before, with WebAssembly (and Rust being having the best toolchain for it) there is a clear route to achieve this, but it will not replace all the software that already exist and can benefit from FFI and C compatibility. Bringing together developers from the many language specific binding generators (helix, neon, rustler, PyO3...) and figuring out what's missing from them (or what is the common parts that can be shared) also seems productive.

• Interoperability with new and unexplored domains. I think Rust benefits enormously from not focusing only in one domain, and choosing to prioritize CLI, WebAssembly, Networking and Embedded is a good subset to start tackling problems, but how to guide other domains to also use Rust and come up with new contributors and expose missing pieces of the larger picture?

Another point extremely close to interoperability is training. A great way to interoperate with other languages and domains is having good documentation and material from transitioning into Rust without having to figure everything at once. Rust documentation is already amazing, especially considering the many books published by each working group. But... there is a gap on the transitions, both from understanding the basics of the language and using it, to the progression from beginner to intermediate and expert.

I see good resources for JavaScript and Python developers, but we are still covering a pretty small niche: programmers curious enough to go learn another language, or looking for solutions for problems in their current language.

Can we bring more people into Rust? RustBridge is obviously the reference here, but there is space for much, much more. Using Rust in The Carpentries lessons? Creating RustOpenSci, mirroring the communities of practice of rOpenSci and pyOpenSci?

Comments?

• Thread on Mastodon
• Thread on Twitter

Neste tutorial, será abordado o processo de criação de um dict ou dicionário, a partir de um ou mais dicts em Python.

Como já é de costume da linguagem, isso pode ser feito de várias maneiras diferentes.

Abordagem inicial

Pra começar, vamos supor que temos os seguintes dicionários:

dict_1 = {
    'a': 1,
    'b': 2,
}

dict_2 = {
    'b': 3,
    'c': 4,
}

Como exemplo, vamos criar um novo dicionário chamado new_dict com os valores de dict_1 e dict_2 logo acima. Uma abordagem bem conhecida é utilizar o método update.

new_dict = {}

new_dcit.update(dict_1)
new_dcit.update(dict_2)

Assim, temos que new_dict será:

>> print(new_dict)
{
    'a': 1,
    'b': 3,
    'c': 4,
}

Este método funciona bem, porém temos de chamar o método update para cada dict que desejamos mesclar em new_dict. Não seria interessante se fosse possível passar todos os dicts necessários já na inicialização de new_dict?

Novidades do Python 3

O Python 3 introduziu uma maneira bem interessante de se fazer isso, utilizando os operadores **.

new_dict = {
    **dict_1,
    **dict_2,
}

Assim, de maneira semelhante ao exemplo anterior, temos que new_dict será :

>> print(new_dict['a'])
1
>> print(new_dict['b'])
3
>> print(new_dict['c'])
4

Cópia real de dicts

Ao utilizamos o procedimento de inicialização acima, devemos tomar conseiderar alguns fatores. Apenas os valores do primeiro nível serão realmente duplicados no novo dicionário. Como exemplo, vamos alterar uma chave presente em ambos os dicts e verificar se as mesmas possuem o mesmo valor:

>> dict_1['a'] = 10
>> new_dict['a'] = 11
>> print(dict_1['a'])
10
>> print(new_dict['a'])
11

Porém isso muda quando um dos valores de dict_1 for uma list, outro dict ou algum objeto complexo. Por exemplo:

dict_3 = {
    'a': 1,
    'b': 2,
    'c': {
        'd': 5,
    }
}

e agora, vamos criar um novo dict a partir desse:

new_dict = {
    **dict_3,
}

Como no exemplo anterior, podemos imaginar que foi realizado uma cópia de todos os elementos de dict_3, porém isso não é totalmente verdade. O que realmente aconteceu é que foi feita uma cópia superficial dos valores de dict_3, ou seja, apenas os valores de primeiro nível foram duplicados. Observe o que acontece quando alteramos o valor do dict presente na chave c.

>> new_dict['c']['d'] = 11
>> print(new_dict['c']['d'])
11
>> print(dict_3['c']['d'])
11 
# valor anterior era 5

No caso da chave c, ela contem uma referência para outra estrutura de dados (um dict, no caso). Quando alteramos algum valor de dict_3['c'], isso reflete em todos os dict que foram inicializados com dict_3. Em outras palavras, deve-se ter cuidado ao inicializar um dict a partir de outros dicts quando os mesmos possuírem valores complexos, como list, dict ou outros objetos (os atributos deste objeto não serão duplicados).

De modo a contornar este inconveniente, podemos utilizar o método deepcopy da lib nativa copy. Agora, ao inicializarmos new_dict:

import copy

dict_3 = {
    'a': 1,
    'b': 2,
    'c': {
        'd': 5,
    }
}

new_dict = copy.deepcopy(dict_3)

O método deepcopy realiza uma cópia recursiva de cada elemento de dict_3, resolvendo nosso problema. Veja mais um exemplo:

>> new_dict['c']['d'] = 11
>> print(new_dict['c']['d'])
11
>> print(dict_3['c']['d'])
5 
# valor não foi alterado

Conclusão

Este artigo tenta demonstrar de maneira simples a criação de dicts, utilizando os diversos recursos que a linguagem oferece bem como os prós e contras de cada abordagem.

Referências

Para mais detalhes e outros exemplos, deem uma olhada neste post do forum da Python Brasil aqui.

É isso pessoal. Obrigado por ler!

Eu e o prof. Leonardo Cruz da Faculdade de Ciências Sociais estamos juntos trabalhando no desenvolvimento do Laboratório Amazônico de Estudos Sociotécnicos da UFPA.

Nossa proposta é realizar leituras e debates críticos sobre o tema da sociologia da tecnologia, produzir pesquisas teóricas e empíricas na região amazônica sobre as relações entre tecnologia e sociedade, e trabalhar com tecnologias livres em comunidades próximas a Belém.

No momento estamos com um grupo de estudos montado com cronograma de textos e filmes para trabalharmos e debatermos criticamente. Esse grupo será o embrião para a orientação de alunos de graduação e pós em temas como impacto da inteligência artificial, computação e guerra, cibernética, vigilantismo, capitalismo de plataforma, fake news, pirataria, software livre, e outros.

Aos interessados, nosso cronograma de estudos está disponível nesse link.

E para quem usa Telegram, pode acessar o grupo de discussão aqui.

Quaisquer dúvidas, só entrar em contato!

Este tutorial é baseado no Intro to Django que fica na parte de baixo da página start do Django project.

Até a data deste post o Django está na versão 2.2.2, e requer Python 3.

O que você precisa?

Python 3.6 ou superior, pip e virtualenv.

Considere que você tenha instalado Python 3.6 ou superior, pip e virtualenv.

Criando o ambiente

Crie uma pasta com o nome django2-pythonclub

$ mkdir django2-pythonclub
$ cd django2-pythonclub

A partir de agora vamos considerar esta como a nossa pasta principal.

Considerando que você está usando Python 3, digite

python3 -m venv .venv

Lembre-se de colocar esta pasta no seu .gitignore, caso esteja usando.

echo .venv >> .gitignore

Depois ative o ambiente digitando

source .venv/bin/activate

Lembre-se, sempre quando você for mexer no projeto, tenha certeza de ter ativado o virtualenv, executando o comando source .venv/bin/activate. Você deve repetir esse comando toda a vez que você abrir um novo terminal.

Instalando Django 2.2.2

Basta digitar

pip install django==2.2.2

Dica: se você digitar pip freeze você verá a versão dos programas instalados.

É recomendável que você atualize a versão do pip

pip install -U pip

Se der erro então faça:

python -m pip install --upgrade pip

Instalando mais dependências

Eu gosto de usar o django-extensions e o django-widget-tweaks, então digite

pip install django-extensions django-widget-tweaks python-decouple

Importante: você precisa criar um arquivo requirements.txt para instalações futuras do projeto em outro lugar.

pip freeze > requirements.txt

Este é o resultado do meu até o dia deste post:

(.venv):$ cat requirements.txt 

django-extensions==2.1.6
django-widget-tweaks==1.4.3
python-decouple==3.1
pytz==2018.9
six==1.12.0

Escondendo a SECRET_KEY e trabalhando com variáveis de ambiente

É muito importante que você não deixe sua SECRET_KEY exposta. Então remova-o imediatamente do seu settings.py ANTES mesmo do primeiro commit. Espero que você esteja usando Git.

Vamos usar o python-decouple escrito por Henrique Bastos para gerenciar nossas variáveis de ambiente. Repare que já instalamos ele logo acima.

Em seguida você vai precisar criar um arquivo .env, para isso rode o comando a seguir, ele vai criar uma pasta contrib e dentro dele colocar um arquivo env_gen.py

if [ ! -d contrib ]; then mkdir contrib; fi; git clone https://gist.github.com/22626de522f5c045bc63acdb8fe67b24.git contrib/
rm -rf contrib/.git/  # remova a pasta .git que está dentro de contrib.

Em seguida rode

python contrib/env_gen.py

que ele vai criar o arquivo .env.

Supondo que você está versionando seu código com Git, é importante que você escreva isso dentro do seu arquivo .gitignore, faça direto pelo terminal

echo .env >> .gitignore
echo .venv >> .gitignore
echo '*.sqlite3' >> .gitignore

Pronto, agora você pode dar o primeiro commit.

Criando o projeto e a App

Para criar o projeto digite

$ django-admin startproject myproject .

repare no ponto no final do comando, isto permite que o arquivo manage.py fique nesta mesma pasta django2-pythonclub .

Agora vamos criar a app bands, mas vamos deixar esta app dentro da pasta myproject. Então entre na pasta

$ cd myproject

e digite

$ python ../manage.py startapp bands

A intenção é que os arquivos tenham a seguinte hierarquia nas pastas:

.
├── manage.py
├── myproject
│   ├── bands
│   │   ├── admin.py
│   │   ├── apps.py
│   │   ├── models.py
│   │   ├── tests.py
│   │   └── views.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
└── requirements.txt

Agora permaneça sempre na pasta django2-pythonclub

cd ..

e digite

$ python manage.py migrate

para criar a primeira migração (isto cria o banco de dados SQLite), e depois rode a aplicação com

$ python manage.py runserver

e veja que a aplicação já está funcionando. Veja o endereço da url aqui

Django version 2.2.2, using settings 'myproject.settings'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

Editando settings.py

Em INSTALLED_APPS acrescente as linhas abaixo.

INSTALLED_APPS = (
    ...
    'widget_tweaks',
    'django_extensions',
    'myproject.bands',
)

E mude também o idioma.

LANGUAGE_CODE = 'pt-br'

E caso você queira o mesmo horário de Brasília-BR

TIME_ZONE = 'America/Sao_Paulo'

Já que falamos do python-decouple, precisamos de mais alguns ajustes

from decouple import config, Csv

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = config('SECRET_KEY')

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = config('DEBUG', default=False, cast=bool)

ALLOWED_HOSTS = config('ALLOWED_HOSTS', default=[], cast=Csv())

Veja que é importante manter sua SECRET_KEY bem guardada (em outro lugar).

Então crie um arquivo .env e guarde sua SECRET_KEY dentro dele, exemplo:

SECRET_KEY=your_secret_key
DEBUG=True
ALLOWED_HOSTS=127.0.0.1,.localhost

Editando models.py

from django.db import models
from django.urls import reverse_lazy


class Band(models.Model):

    """A model of a rock band."""
    name = models.CharField(max_length=200)
    can_rock = models.BooleanField(default=True)

    class Meta:
        ordering = ('name',)
        verbose_name = 'band'
        verbose_name_plural = 'bands'

    def __str__(self):
        return self.name

    def get_absolute_url(self):
        # retorna a url no formato /bands/1/
        return reverse_lazy('band_detail', kwargs={'pk': self.pk})

    def get_members_count(self):
        # count members by band
        # conta os membros por banda
        return self.band.count()


class Member(models.Model):

    """A model of a rock band member."""
    name = models.CharField("Member's name", max_length=200)
    instrument = models.CharField(choices=(
        ('g', "Guitar"),
        ('b', "Bass"),
        ('d', "Drums"),
        ('v', "Vocal"),
        ('p', "Piano"),
    ),
        max_length=1
    )

    band = models.ForeignKey("Band", related_name='band', on_delete=models.CASCADE)

    class Meta:
        ordering = ('name',)
        verbose_name = 'member'
        verbose_name_plural = 'members'

    def __str__(self):
        return self.name

Tem algumas coisas que eu não estou explicando aqui para o tutorial ficar curto, mas uma coisa importante é que, como nós editamos o models.py vamos precisar criar um arquivo de migração do novo modelo. Para isso digite

python manage.py makemigrations
python manage.py migrate

O primeiro comando cria o arquivo de migração e o segundo o executa, criando as tabelas no banco de dados.

Editando urls.py

from django.urls import include, path
from myproject.bands import views as v
from django.contrib import admin

app_name = 'bands'

urlpatterns = [
    path('', v.home, name='home'),
    # path('bands/', v.band_list, name='bands'),
    # path('bands/<int:pk>/', v.band_detail, name='band_detail'),
    # path('bandform/', v.BandCreate.as_view(), name='band_form'),
    # path('memberform/', v.MemberCreate.as_view(), name='member_form'),
    # path('contact/', v.band_contact, name='contact'),
    # path('protected/', v.protected_view, name='protected'),
    # path('accounts/login/', v.message),
    path('admin/', admin.site.urls),
]

Obs: deixei as demais urls comentada porque precisa da função em views.py para que cada url funcione. Descomente cada url somente depois que você tiver definido a função em classe em views.py a seguir.

Editando views.py

from django.shortcuts import render
from django.http import HttpResponse
from django.contrib.auth.decorators import login_required
from django.views.generic import CreateView
from django.urls import reverse_lazy
from .models import Band, Member
# from .forms import BandContactForm, BandForm, MemberForm

Obs: Deixei a última linha comentada porque ainda não chegamos em forms.

A função a seguir retorna um HttpResponse, ou seja, uma mensagem simples no navegador.

def home(request):
    return HttpResponse('Welcome to the site!')

A próxima função (use uma ou outra) renderiza um template, uma página html no navegador.

def home(request):
    return render(request, 'home.html')

A função band_list retorna todas as bandas.

Para fazer a busca por nome de banda usamos o comando search = request.GET.get('search_box'), onde search_box é o nome do campo no template band_list.html.

E os nomes são retornados a partir do comando bands = bands.filter(name__icontains=search). Onde icontains procura um texto que contém a palavra, ou seja, você pode digitar o nome incompleto (ignora maiúsculo ou minúsculo).

def band_list(request):
    """ A view of all bands. """
    bands = Band.objects.all()
    search = request.GET.get('search_box')
    if search:
        bands = bands.filter(name__icontains=search)
    return render(request, 'bands/band_list.html', {'bands': bands})

Em urls.py pode descomentar a linha a seguir:

path('bands/', v.band_list, name='bands'),

A função band_contact mostra como tratar um formulário na view. Esta função requer BandContactForm, explicado em forms.py.

def band_contact(request):
    """ A example of form """
    if request.method == 'POST':
        form = BandContactForm(request.POST)
    else:
        form = BandContactForm()
    return render(request, 'bands/band_contact.html', {'form': form})

Em urls.py pode descomentar a linha a seguir:

path('contact/', v.band_contact, name='contact'),

A função band_detail retorna todos os membros de cada banda, usando o pk da banda junto com o comando filter em members.

def band_detail(request, pk):
    """ A view of all members by bands. """
    band = Band.objects.get(pk=pk)
    members = Member.objects.all().filter(band=band)
    context = {'members': members, 'band': band}
    return render(request, 'bands/band_detail.html', context)

Em urls.py pode descomentar a linha a seguir:

path('bands/<int:pk>/', v.band_detail, name='band_detail'),

BandCreate e MemberCreate usam o Class Based View para tratar formulário de uma forma mais simplificada usando a classe CreateView. O reverse_lazy serve para tratar a url de retorno de página.

As classes a seguir requerem BandForm e MemberForm, explicado em forms.py.

class BandCreate(CreateView):
    model = Band
    form_class = BandForm
    template_name = 'bands/band_form.html'
    success_url = reverse_lazy('bands')


class MemberCreate(CreateView):
    model = Member
    form_class = MemberForm
    template_name = 'bands/member_form.html'
    success_url = reverse_lazy('bands')

Em urls.py pode descomentar a linha a seguir:

path('bandform/', v.BandCreate.as_view(), name='band_form'),
path('memberform/', v.MemberCreate.as_view(), name='member_form'),

A próxima função requer que você entre numa página somente quando estiver logado.

[@login_required](https://docs.djangoproject.com/en/2.2/topics/auth/default/#the-login-required-decorator) é um decorator.

login_url='/accounts/login/' é página de erro, ou seja, quando o usuário não conseguiu logar.

E render(request, 'bands/protected.html',... é página de sucesso.

@login_required(login_url='/accounts/login/')
def protected_view(request):
    """ A view that can only be accessed by logged-in users """
    return render(request, 'bands/protected.html', {'current_user': request.user})

HttpResponse retorna uma mensagem simples no navegador sem a necessidade de um template.

def message(request):
    """ Message if is not authenticated. Simple view! """
    return HttpResponse('Access denied!')

Em urls.py pode descomentar a linha a seguir:

path('protected/', v.protected_view, name='protected'),
path('accounts/login/', v.message),

Comandos básicos do manage.py

Para criar novas migrações com base nas alterações feitas nos seus modelos

$ python manage.py makemigrations bands

Obs: talvez dê erro porque está faltando coisas de forms.py, explicado mais abaixo.

Para aplicar as migrações

$ python manage.py migrate

Para criar um usuário e senha para o admin

$ python manage.py createsuperuser

Para rodar a aplicação localmente

$ python manage.py runserver

Após criar um super usuário você pode entrar em localhost:8000/admin

Obs: Se você entrar agora em localhost:8000 vai faltar o template home.html. Explicado mais abaixo.

shell_plus

É o interpretador interativo do python rodando via terminal direto na aplicação do django.

Com o comando a seguir abrimos o shell do Django.

$ python manage.py shell

Mas se você está usando o django-extensions (mostrei como configurá-lo no settings.py), então basta digitar

$ python manage.py shell_plus

Veja a seguir como inserir dados direto pelo shell.

>>> from myproject.bands.models import Band, Member
>>> # Com django-extensions não precisa fazer o import
>>> # criando o objeto e salvando
>>> band = Band.objects.create(name='Metallica')
>>> band.name
>>> band.can_rock
>>> band.id
>>> # criando uma instancia da banda a partir do id
>>> b = Band.objects.get(id=band.id)
>>> # criando uma instancia do Membro e associando o id da banda a ela
>>> m = Member(name='James Hetfield', instrument='b', band=b)
>>> m.name
>>> # retornando o instrumento
>>> m.instrument
>>> m.get_instrument_display()
>>> m.band
>>> # salvando
>>> m.save()
>>> # listando todas as bandas
>>> Band.objects.all()
>>> # listando todos os membros
>>> Member.objects.all()
>>> # criando mais uma banda
>>> band = Band.objects.create(name='The Beatles')
>>> band = Band.objects.get(name='The Beatles')
>>> band.id
>>> b = Band.objects.get(id=band.id)
>>> # criando mais um membro
>>> m = Member(name='John Lennon', instrument='v', band=b)
>>> m.save()
>>> # listando tudo novamente
>>> Band.objects.all()
>>> Member.objects.all()
>>> exit()

Criando os templates

Você pode criar os templates com os comandos a seguir...

$ mkdir -p myproject/bands/templates/bands
$ touch myproject/bands/templates/{menu,base,home}.html
$ touch myproject/bands/templates/bands/{band_list,band_detail,band_form,band_contact,member_form,protected}.html

... ou pegar os templates já prontos direto do Github.

mkdir -p myproject/bands/templates/bands
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/base.html -P myproject/bands/templates/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/home.html -P myproject/bands/templates/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/menu.html -P myproject/bands/templates/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/band_contact.html -P myproject/bands/templates/bands/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/band_detail.html -P myproject/bands/templates/bands/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/band_form.html -P myproject/bands/templates/bands/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/band_list.html -P myproject/bands/templates/bands/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/member_form.html -P myproject/bands/templates/bands/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/myproject/bands/templates/bands/protected.html -P myproject/bands/templates/bands/

forms.py

$ touch myproject/bands/forms.py

Edite o forms.py.

from django import forms
from .models import Band, Member


class BandContactForm(forms.Form):
    subject = forms.CharField(max_length=100)
    message = forms.CharField(widget=forms.Textarea)
    sender = forms.EmailField()
    cc_myself = forms.BooleanField(required=False)


class BandForm(forms.ModelForm):

    class Meta:
        model = Band
        fields = '__all__'


class MemberForm(forms.ModelForm):

    class Meta:
        model = Member
        fields = '__all__'

Lembra que eu deixei o código comentado em views.py?

Descomente ele por favor

from .forms import BandContactForm, BandForm, MemberForm

admin.py

Criamos uma customização para o admin onde em members aparece um filtro por bandas.

from django.contrib import admin
from .models import Band, Member


class MemberAdmin(admin.ModelAdmin):
    """Customize the look of the auto-generated admin for the Member model."""
    list_display = ('name', 'instrument')
    list_filter = ('band',)


admin.site.register(Band)  # Use the default options
admin.site.register(Member, MemberAdmin)  # Use the customized options

Carregando dados de um CSV

Vamos baixar alguns arquivos para criar os dados no banco a partir de um CSV.

wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/create_data.py
mkdir fix
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/fix/bands.csv -P fix/
wget https://raw.githubusercontent.com/rg3915/django2-pythonclub/master/fix/members.csv -P fix/

Estando na pasta principal, rode o comando

python create_data.py

que ele vai carregar alguns dados pra você.

Veja o código de create_data.py.

Veja o código completo em https://github.com/rg3915/django2-pythonclub

git clone https://github.com/rg3915/django2-pythonclub.git

Often when starting a new Python project we need to spend some time thinking about how to manage the settings, decide on which module the configuration manager will be written, decide which name to give to this module, create a class or function to store the configuration keys, create the conditions for multiple environments and still need to worry about where these keys will be stored and in which file format?

No more! now you have Dynaconf!

Spend your precious time developing your application, run pip install dynaconf and let Dynaconf take care of your settings.

Quick start.

from dynaconf import settings

And that's it!

That is the only line of code you need, no complicated boilerplate, no hadouken-ifs, no need to maintain config classes.

You must be wondering - "What magic is this? Where does the setting values come from?"

Well, there is no magic, and the values can come from wherever you want, by default and following the recommendations of the 12 factor apps Dynaconf has preference for environment variables.

# optionally you can save it in .env file
export DYNACONF_DEBUG=true
export DYNACONF_NAME=Bruno

# app.py
from dynaconf import settings
if settings.DEBUG is True:
    print(settings.NAME)

$ python3 app.py
Bruno

And the environment variables for Dynaconf are typed using the toml format sotrue has been evaluated to boolean True and this makes it possible to export lists, dictionaries, floats, booleans, and so on.

Read more about envvars

More than environment variables

Well, that's cool, but your project will not have settings coming from just the environment variables, I'm sure you want to have a settings file where you can set default values.

Dynaconf can read multiple file formats, out of the box it supports .py, .toml, .ini and .json. If PyYAML is installed then it will also support .yaml and you don't have to take care of finding and opening the files. The preferred format is .toml because it is currently the best configuration format, widely addopted, and you can use whatever file format you want.

# settings.toml
[default]
name = "Bruno"

[development]
debug = true

[production]
debug = false

# app.py
from dynaconf import settings
if settings.DEBUG is True:
    print(settings.NAME)

$ python3 app.py
Bruno

And as you can see now using settings. file we can have separate [environments] by default dynaconf will always work on [development] which means only [default] and [development] variables will be loaded. At any time you can do export ENV_FOR_DYNACONF=production and then it starts using the values from [production] environment.

If you don't want to have that separation by environment, you can simply put everything under [default] section.

Read more about environments and settings file

Some values are secrets

A good practice is to not store your secrets like passwords and tokens directly on settings files, because you can make a mistake and commit that to a public git repository, so there are some alternatives to store secrets

Environment Variables

Not recommended

There are some people who disagrees and it is really a point of security failure. However, if you are sure that your machine is protected, you can leave the secrets in the variables, at your own risk, Dynaconf can read it normally.

Secret files

This is a simple level of security for keeping secrets, and it is specially useful to keep development secrets. That token you use to access the development API etc.

It is very simple, together with your normal settings.toml you put a new file called .secrets.toml and store your sensitive data there. Dynaconf will read it after the read of the settings.toml

Wait.. how does it solve my security problem?

Well it does not (yet) but it make your life easier in 2 ways.

1. Put .secrets.* in your ~/.gitignore so you will never commit the mistake of sending that data to a public git repository.
2. Dynaconf can output debug information when DEBUG_LEVEL_FOR_DYNACON=DEBUG is exported, all loaded values are printed but if the values comes from a .secrets.* file, then only the key is printed and the value is masked. It is useful to use on public CI.
3. You can setup a step on your Ansible deployment playbook that will safely copy or generate the secrets file to store there on your production environment.

You can also tell Dynaconf to load that file from somewhere else export SECRETS_FOR_DYNACONF=/path/to/secrets/location/.secrets.yaml (very useful for CI like Jenkins)

Vault Project from Hashicorp

Recommended!

Now we are really talking about true security

Using Vault is the better way to protect your secrets dynaconf has built-in support:

export VAULT_ENABLED_FOR_DYNACONF=true
export VAULT_URL_FOR_DYNACONF=https://..../
export OTHER_VAULT_CONFIGS

And then if have for example the TOKEN stores on your vault server you can simply do:

from dynaconf import settings
perform_your_authentication(settings.TOKEN)

Vault has lots of features like leases and sealed vaults.

Read More

Are you using Django or Flask?

Dynaconf provides extensions for those 2 frameworks, with 2 lines of code you enable it and then your framework will start reading settings from Dynaconf.

Django

# settings.py
import dynaconf  # noqa
settings = dynaconf.DjangoDynaconf(__name__, **options)

Now you if you do export DJANGO_FOO=BAR you can access inside your app via django.conf.settings.FOO

read more

Flask

# app.py
from dynaconf import FlaskDynaconf
FlaskDynaconf(app, **options)

Now you if you do export FLASK_FOO=BAR you can access inside your app via app.config['FOO']

read more

What if you are using a different settings file format? a different framework and a different external storage?

You can extend Dynaconf adding new loaders!

Dynaconf already provides loaders for:

• .py
• .json
• .yaml
• .toml
• .ini
• Redis Server
• Vault Server
• .env files
• Environment variables

But if this is not a fit for your project you can still create your own loader

Conclusion

Dynaconf is the only thing you need to manage your settings!

• Well tested
• Trusted by companies like Red Hat, Seek, Catho and others
• Well tested both on Linux and Windows environment
• Strict separation of settings from code (following 12-factor applications Guide).
• Define comprehensive default values.
• Store parameters in multiple file formats (.toml, .json, .yaml, .ini and .py).
• Sensitive secrets like tokens and passwords can be stored in safe places like .secrets file or vault server.
• Parameters can optionally be stored in external services like Redis server.
• Simple feature flag system.
• Layered [environment] system.
• Environment variables can be used to override parameters.
• Support for .env files to automate the export of environment variables.
• Correct data types (even for environment variables).
• Have only one canonical settings module to rule all your instances.
• Drop in extension for Flask app.config object.
• Drop in extension for Django conf.settings object.
• Powerful $ dynaconf CLI to help you manage your settings via console.
• Customizable Validation System to ensure correct config parameters.
• Allow the change of dynamic parameters on the fly without the need to redeploy your application.

Read the docs

Settings are simple but Dynaconf provides even more features like Feature Flags, Settings Context Managers, Plugin Settings etc..

Documentation: http://dynaconf.readthedocs.io/

Dynaconf is waiting for your feedback and Contribution

:)

from dynaconf import settings
settings.THANKS_FOR_READING