Planeta PythonBrasilhttp://planet.python.org.br/atom.xml2021-07-12T18:01:03+00:00Planet/2.0 +http://www.planetplanet.org7 subjects (and GitHub repositories) to become a better Go Developerhttps://avelino.run/7-subjects-and-github-repositories-to-become-a-better-go-developer/2021-07-11T00:00:00+00:00With the high adoption of the Go language by developers and large companies, this has led companies to search for engineers with experience in Go.
This can create a lot of pressure of what to study to become a better engineer, this is very personal, it requires planning of what and when to study other subjects (even outside the engineering area).
In this blogpost some topics (with repositories and links) that I think are important to know in order to become an engineer person with even better Go knowledge, follow good practices for writing code, concepts of code structure (usually using design pattern), scalable code and clean code.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/B9wLANsM1rE" width="1" />Thiago Avelinohttps://avelino.run/blog/Ciclo de Entrevistas sobre as Pesquisas no PPGCC da UFPA – Inteligência Computacionalhttps://blog.filipesaraiva.info/?p=22502021-06-21T21:51:57+00:00<p><img alt="" class="aligncenter wp-image-2251" height="711" src="https://blog.filipesaraiva.info/wp-content/uploads/2021/06/facomp-podcast-1024x1024.jpg" width="711" /></p>
<p>A <a href="http://computacao.ufpa.br/">Faculdade de Computação</a> e o <a href="https://ppgcc.propesp.ufpa.br/">Programa de Pós-Graduação em Ciência da Computação</a> da <a href="https://ufpa.br/">UFPA</a> estão desenvolvendo um projeto que pretende atingir dois objetivos: o primeiro, fazer uma melhor divulgação para o público externo à universidade do que produzimos em nossas pesquisas; o segundo, uma melhor divulgação INTERNA da mesma coisa – o que desenvolvemos em nossas pesquisas.</p>
<p>Sim, <strong>INTERNA</strong> – não bastasse de fato a comunicação deficitária e pulverizada do que fazemos para os nossos próprios alunos, a pandemia só veio a piorar esse quadro. Após mais de um ano sem contato próximo com as turmas, com aulas completamente à distância e sem maiores interações extra-classe, os alunos em geral estão perdidos sobre as possibilidades de temas para TCCs e pesquisas que eles podem realizar conosco.</p>
<p>Dessa forma as duas subunidades estão entrevistando todos os professores para que falem um pouco sobre os temas que trabalham, o histórico de pesquisa, o que vem sendo feito e, mais interessante, o que pode vir a ser feito. As entrevistas ocorrem no canal do YouTube <a href="https://www.youtube.com/c/ComputaçãoUFPA/">Computação UFPA</a> e depois são retrabalhadas para aparecerem no <a href="https://open.spotify.com/show/7oGkFKS4eV4WM2XjbEFeJu">FacompCast</a>.</p>
<p>Feitas as devidas introduções, nesta terça dia 22/06 às 11h eu e o amigo prof. Jefferson Morais iremos falar um pouco sobre as pesquisas em Inteligência Computacional (ou seria Artificial?) desenvolvidas por nós. Será um bom apanhado sobre os trabalhos em 4 áreas que atuamos – aprendizado de máquina, metaheurísticas, sistemas fuzzy e sistemas multiagentes -, expondo projetos atuais e novos para os interessados.</p>
<p>Nos vemos portanto logo mais na <a href="https://www.youtube.com/watch?v=VRytpiFWTQI">sala da entrevista</a>.</p>
<p><strong>UPDATE:</strong></p>
<p>A gravação já está disponível, segue abaixo:</p>
<p></p>Filipe Saraivahttps://blog.filipesaraiva.infoDecisões na carreira de engenharia de software (desenvolvimento)https://avelino.run/decisoes-na-carreira-de-engenharia-de-software-desenvolvimento/2021-06-18T00:00:00+00:00Ao decidir sobre qual passo seguirá em sua trajetória profissional, acredito que vale pena considerar os seguintes fatores (1). Dependendo do seu momento de pessoal e/ou profissional, diferentes fatores será mais (ou menos) importantes (2).
Importante:
Assumi-se que você achou melhor se juntar, ao invés de criar (começar uma empresa). Não estou falando de fundar ou não uma empresa, e sim desmitificar auxiliar que queira entrar em uma empresa existente e é parcial com pessoas no início da carreira.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/AZ3KwojvUaI" width="1" />Thiago Avelinohttps://avelino.run/blog/Accessing Google Firestore on Vercelhttps://nullonerror.org/2021/06/14/accessing-google-firestore-on-vercel2021-06-14T00:00:00+00:00<p>Or on any other cloud service, or language.</p>
<blockquote>
<p>TL;DR: Use GOOGLE_APPLICATION_CREDENTIALS with a valid JSON credential to use any Google APIs anywhere.</p>
</blockquote>
<p><a href="https://firebase.google.com/docs/hosting">Firebase Hosting</a> is great, but the new <a href="https://vercel.com/">Vercel</a> is awesome for NextJS apps. On Vercel your code runs on <a href="https://aws.amazon.com/lambda/edge/">Lambda@Edge</a> and it is cached on <a href="https://aws.amazon.com/cloudfront/">CloudFront</a>; in the same way, Firebase uses <a href="https://www.fastly.com/">Fastly</a>, another great CDN.</p>
<p>You can not take full advantage of running a NextJS app on Firebase Hosting, only on Vercel, or by deploying manually.</p>
<p>I like to use <a href="https://firebase.google.com/docs/firestore">Firestore</a> on some projects, and unfortunately it is “restricted” to the internal network of Google Cloud, although there is a trick; you can download the service account and export an environment variable named <code class="language-plaintext highlighter-rouge">GOOGLE_APPLICATION_CREDENTIALS</code> with the path of the downloaded credential.</p>
<p>First, download the JSON file following <a href="https://firebase.google.com/docs/admin/setup#initialize-sdk">this steps</a>.</p>
<p>Then, convert the credentials JSON file to <em>base64</em>:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> ~/Downloads/project-name-adminsdk-owd8n-43fca28a2a.json | <span class="nb">base64</span>
</code></pre></div></div>
<p>Now copy the result and create an <a href="https://vercel.com/docs/environment-variables">environment variable on Vercel</a> named <code class="language-plaintext highlighter-rouge">GOOGLE_CREDENTIALS</code> and paste the contents.</p>
<p>On your NextJS project, create a <code class="language-plaintext highlighter-rouge">pages/api/function.js</code> and add the following code:</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">os</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">os</span><span class="dl">"</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">promises</span> <span class="k">as</span> <span class="nx">fsp</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">fs</span><span class="dl">"</span>
<span class="k">import</span> <span class="nx">path</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">path</span><span class="dl">"</span>
<span class="k">import</span> <span class="p">{</span> <span class="nx">Firestore</span><span class="p">,</span> <span class="nx">FieldValue</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@google-cloud/firestore</span><span class="dl">"</span>
<span class="kd">let</span> <span class="nx">_firestore</span> <span class="o">=</span> <span class="kc">null</span>
<span class="kd">const</span> <span class="nx">lazyFirestore</span> <span class="o">=</span> <span class="p">()</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="o">!</span><span class="nx">_firestore</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">baseDir</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">fsp</span><span class="p">.</span><span class="nx">mkdtemp</span><span class="p">((</span><span class="k">await</span> <span class="nx">fsp</span><span class="p">.</span><span class="nx">realpath</span><span class="p">(</span><span class="nx">os</span><span class="p">.</span><span class="nx">tmpdir</span><span class="p">()))</span> <span class="o">+</span> <span class="nx">path</span><span class="p">.</span><span class="nx">sep</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">fileName</span> <span class="o">=</span> <span class="nx">path</span><span class="p">.</span><span class="nx">join</span><span class="p">(</span><span class="nx">baseDir</span><span class="p">,</span> <span class="dl">"</span><span class="s2">credentials.json</span><span class="dl">"</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">buffer</span> <span class="o">=</span> <span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">GOOGLE_CREDENTIALS</span><span class="p">,</span> <span class="dl">"</span><span class="s2">base64</span><span class="dl">"</span><span class="p">)</span>
<span class="k">await</span> <span class="nx">fsp</span><span class="p">.</span><span class="nx">writeFile</span><span class="p">(</span><span class="nx">fileName</span><span class="p">,</span> <span class="nx">buffer</span><span class="p">)</span>
<span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">[</span><span class="dl">"</span><span class="s2">GOOGLE_APPLICATION_CREDENTIALS</span><span class="dl">"</span><span class="p">]</span> <span class="o">=</span> <span class="nx">fileName</span>
<span class="nx">_firestore</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Firestore</span><span class="p">()</span>
<span class="p">}</span>
<span class="k">return</span> <span class="nx">_firestore</span>
<span class="p">}</span>
<span class="k">export</span> <span class="k">default</span> <span class="k">async</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">firestore</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">lazyFirestore</span><span class="p">()</span>
<span class="kd">const</span> <span class="nx">increment</span> <span class="o">=</span> <span class="nx">FieldValue</span><span class="p">.</span><span class="nx">increment</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
<span class="kd">const</span> <span class="nx">documentRef</span> <span class="o">=</span> <span class="nx">firestore</span><span class="p">.</span><span class="nx">collection</span><span class="p">(</span><span class="dl">"</span><span class="s2">v1</span><span class="dl">"</span><span class="p">).</span><span class="nx">doc</span><span class="p">(</span><span class="dl">"</span><span class="s2">default</span><span class="dl">"</span><span class="p">)</span>
<span class="k">await</span> <span class="nx">documentRef</span><span class="p">.</span><span class="nx">update</span><span class="p">({</span> <span class="na">counter</span><span class="p">:</span> <span class="nx">increment</span> <span class="p">})</span>
<span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">200</span><span class="p">).</span><span class="nx">json</span><span class="p">({})</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Done! Now it is possible to use Firestore on Vercel or anywhere.</p>
<p><a href="https://github.com/skhaz/firestore-on-vercel">Project of example</a>.</p><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/NullOnError?a=1qzG_tkTilY:N0MeQSGkJJg:yIl2AUoC8zA"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=yIl2AUoC8zA" /></a> <a href="http://feeds.feedburner.com/~ff/NullOnError?a=1qzG_tkTilY:N0MeQSGkJJg:qj6IDK7rITs"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=qj6IDK7rITs" /></a>
</div><img alt="" height="1" src="http://feeds.feedburner.com/~r/NullOnError/~4/1qzG_tkTilY" width="1" />Or on any other cloud service, or language.Rodrigo Delducarodrigodelduca@gmail.comhttps://nullonerror.org/Reconhecimento pelo trabalho com Open Source - GitHub Starhttps://avelino.run/reconhecimento-pelo-trabalho-com-open-source-github-star/2021-06-10T00:00:00+00:00Tenho a felicidade de compartilhar que fui premiado com o status de GitHub Stars por GitHub e nomeação por diversas pessoas - obrigado a todos, que tornaram isso possível.
Estou contribuindo e criando software Open Source desde 2008. Entrei no GitHub 31 outubro de 2008 e vê um universo de oportunidade para aprender olhando código de softwares que usava no meu dia a dia. No começo tive dificuldade de receber não em minhas contribuições (issues e pull requests), mas depois que entendi a “dinâmica” de um projeto open source comecei encarar o não como oportunidade de aprender.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/YxrIxM5CfE4" width="1" />Thiago Avelinohttps://avelino.run/blog/Minha meta como gestor é: ser mandado embora no final do diahttps://avelino.run/minha-meta-como-gestor-eh-ser-mandado-embora-no-final-do-dia/2021-05-19T00:00:00+00:00Compartilho com meu time que minha meta é ser “mandado em bora” no final do dia, o que quero dizer com isso?
Meu trabalho como gestor é fazer meu time trabalhar sem minha dependência, se eles estão conseguindo andar (entregar o combinado, dar suporte a profissionais menos experientes, se comunicar com pessoas não técnicas e etc) sem minha dependência quer dizer que fiz um ótimo trabalho dando a autonomia necessária para todos tomar decisão sem pedir permissão.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/jb-1TrWD0NA" width="1" />Thiago Avelinohttps://avelino.run/blog/Orientação a objetos de outra forma: Propertytag:pythonclub.com.br,2021-05-17:/oo-de-outra-forma-6.html2021-05-17T21:00:00+00:00<p>Seguindo com a série, chegou a hora de discutir sobre encapsulamento, ou seja, ocultar detalhes de implementação de uma classe do resto do código. Em algumas linguagens de programação isso é feito utilizando <code>protected</code> ou <code>private</code>, e às vezes o acesso aos atributos é feito através de funções <em>getters</em> e <em>setters</em>. Nesse texto vamos ver como o Python lida com essas questões.</p>
<h2>Métodos protegidos e privados</h2>
<p>Diferente de linguagens como Java e PHP que possuem palavras-chave como <code>protected</code> e <code>private</code> para impedir que outras classes acessem determinados métodos ou atributos, Python deixa tudo como público. Porém isso não significa que todas as funções de uma classe podem ser chamadas por outras, ou todos os atributos podem ser lidos e alterados sem cuidados.</p>
<p>Para que quem estiver escrevendo um código saiba quais as funções ou atributos que não deveriam ser acessados diretamente, segue-se o padrão de começá-los com <code>_</code>, de forma similar aos arquivos ocultos em sistemas UNIX, que começam com <code>.</code>. Esse padrão já foi seguido na classe <code>AutenticavelComRegistro</code> da postagem sobre <a href="https://dev.to/acaverna/orientacao-a-objetos-de-outra-forma-heranca-multiplas-e-mixins-31eb">mixins</a>, onde a função que pega a data do sistema foi nomeada <code>_get_data</code>. Entretanto isso é apenas uma sugestão, nada impede dela ser chamada, como no exemplo a baixo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="k">class</span> <span class="nc">Exemplo</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">_get_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s1">'</span><span class="si">%d</span><span class="s1">/%m/%Y %T'</span><span class="p">)</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">Exemplo</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="n">_get_data</span><span class="p">())</span>
</pre></div>
<p>Porém algumas bibliotecas também utilizam o <code>_</code> para indicar outras informações como metadados do objeto, e que podem ser acessados sem muitos problemas. Assim é possível utilizar esse símbolo duas vezes (<code>__</code>) para indicar que realmente essa variável ou função não deveria ser acessada de fora da classe, apresentando erro de que o atributo não foi encontrado ao tentar executar a função, porém ela ainda pode ser acessada:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="k">class</span> <span class="nc">Exemplo</span><span class="p">:</span>
<span class="k">def</span> <span class="nf">__get_data</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s1">'</span><span class="si">%d</span><span class="s1">/%m/%Y %T'</span><span class="p">)</span>
<span class="n">obj</span> <span class="o">=</span> <span class="n">Exemplo</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="n">__get_data</span><span class="p">())</span> <span class="c1"># AttributeError</span>
<span class="nb">print</span><span class="p">(</span><span class="n">obj</span><span class="o">.</span><span class="n">_Exemplo__get_data</span><span class="p">())</span> <span class="c1"># Executa a função</span>
</pre></div>
<h2>Property</h2>
<p>Os <em>getters</em> e <em>setters</em> muitas vezes são usados para impedir que determinadas variáveis sejam alteradas, ou validar o valor antes de atribuir a variável, ou ainda processar um valor a partir de outras variáveis. Porém como o Python incentiva o acesso direto as variáveis, existe a <em>property</em>, que ao tentar acessar uma variável ou alterar um valor, uma função é chamada. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">nome</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_nome</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.nome}</span><span class="s1"> </span><span class="si">{self.sobrenome}</span><span class="s1">'</span>
<span class="nd">@nome_completo</span><span class="o">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="n">valor</span> <span class="o">=</span> <span class="n">valor</span><span class="o">.</span><span class="n">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">,</span> <span class="mi">1</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_nome</span> <span class="o">=</span> <span class="n">valor</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">valor</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">idade</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">_idade</span>
<span class="nd">@idade</span><span class="o">.</span><span class="n">setter</span>
<span class="k">def</span> <span class="nf">idade</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">if</span> <span class="n">valor</span> <span class="o"><</span> <span class="mi">0</span><span class="p">:</span>
<span class="k">raise</span> <span class="ne">ValueError</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_idade</span> <span class="o">=</span> <span class="n">valor</span>
<span class="k">def</span> <span class="nf">fazer_aniversario</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
<p>Nesse código algumas variáveis são acessíveis através de <em>properties</em>, de forma geral, as variáveis foram definidas começando com <code>_</code> e com uma <em>property</em> de mesmo nome (sem o <code>_</code>). O primeiro caso é o <code>nome</code>, que possui apenas o <em>getter</em>, sendo possível o seu acesso como <code>obj.nome</code>, porém ao tentar atribuir um valor, será lançado um erro (<code>AttributeError: can't set attribute</code>). Em relação ao <code>sobrenome</code>, como não é necessário nenhum tratamento especial, não foi utilizado um <em>property</em>, porém futuramente pode ser facilmente substituído por um sem precisar alterar os demais códigos. Porém a função <code>nome_completo</code> foi substituída por um <em>property</em>, permitindo tanto o acesso ao nome completo da pessoa, como se fosse uma variável, quanto trocar <code>nome</code> e <code>sobrenome</code> ao atribuir um novo valor para essa <em>property</em>. Quanto a <code>idade</code> utiliza o <em>setter</em> do <em>property</em> para validar o valor recebido, retornando erro para idades inválidas (negativas).</p>
<p>Vale observar também que todas as funções de <em>getter</em> não recebem nenhum argumento (além do <code>self</code>), enquanto as funções de <em>setter</em> recebem o valor atribuído à variável.</p>
<p>Utilizando a <a href="https://dev.to/acaverna/orientacao-a-objetos-de-outra-forma-abc-89b">ABC</a>, ainda é possível informar que alguma classe filha deverá implementar alguma <em>property</em>. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span>
<span class="k">class</span> <span class="nc">Pessoa</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="nd">@property</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">Brasileiro</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.nome}</span><span class="s1"> </span><span class="si">{self.sobrenome}</span><span class="s1">'</span>
<span class="k">class</span> <span class="nc">Japones</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="nd">@property</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.sobrenome}</span><span class="s1"> </span><span class="si">{self.nome}</span><span class="s1">'</span>
</pre></div>
<h2>Considerações</h2>
<p>Diferente de algumas linguagens que ocultam as variáveis dos objetos, permitindo o seu acesso apenas através de funções, Python seguem no sentido contrário, acessando as funções de <em>getter</em> e <em>setter</em> como se fossem variáveis, isso permite começar com uma classe simples e ir adicionando funcionalidades conforme elas forem necessárias, sem precisar mudar o código das demais partes da aplicação, além de deixar transparente para quem desenvolve, não sendo necessário lembrar se precisa usar <em>getteres</em> e <em>setteres</em> ou não.</p>
<p>De forma geral, programação orientada a objetos consiste em seguir determinados padrões de código, e as linguagens que implementam esse paradigma oferecem facilidades para escrever código seguindo esses padrões, e às vezes até ocultando detalhes complexos de suas implementações. Nesse contexto, eu recomendo a palestra do autor do <a href="https://htop.dev/">htop</a> feita no FISL 16, onde ele comenta como usou <a href="http://hemingway.softwarelivre.org/fisl16/high/41f/sala_41f-high-201507091200.ogv">orientação a objetos em C</a>. E para quem ainda quiser se aprofundar no assunto de orientação a objetos no Python, recomendo os vídeos do <a href="https://www.youtube.com/playlist?list=PLOQgLBuj2-3L_L6ahsBVA_SzuGtKre3OK">Eduardo Mendes</a> (também conhecido como dunossauro).</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Iniciando com o ORM Pony no Python III - Erros e Exceçõeshttps://paulohrpinheiro.xyz/texts/python/2021-05-16-iniciando-com-o-orm-pony-no-python-iii-erros-e-excecoes.html2021-05-16T00:00:00+00:00Seguindo o trilha com o ORM Pony, neste terceiro texto, agora começaremos a gerar situações de erro e ver o que os objetos retornam ou que exceções são geradas.Blog do PauloHRPinheirohttps://paulohrpinheiro.xyzDecifrando o Zen do Pythonhttps://paulohrpinheiro.xyz/texts/python/2021-05-12-decifrando-o-zen-do-python.html2021-05-12T00:00:00+00:00Eis que buscando algum conjunto de dados dentro do Python, deparei-me com a verdade sobre o Zen do PythonBlog do PauloHRPinheirohttps://paulohrpinheiro.xyzIniciando com o ORM Pony no Python II - Banco de Dados com Dockerhttps://paulohrpinheiro.xyz/texts/python/2021-05-11-iniciando-com-o-orm-pony-no-python-ii-banco-de-dados-com-docker.html2021-05-11T00:00:00+00:00Continuando a jornada com o ORM Pony, agora com outros Banco de Dados, em DockerBlog do PauloHRPinheirohttps://paulohrpinheiro.xyzOrientação a objetos de outra forma: ABCtag:pythonclub.com.br,2021-05-10:/oo-de-outra-forma-5.html2021-05-10T15:00:00+00:00<p>Na discussão sobre <a href="https://dev.to/acaverna/orientacao-a-objetos-de-outra-forma-heranca-multiplas-e-mixins-31eb">herança e mixins</a> foram criadas várias classes, como <code>Autenticavel</code> e <code>AutenticavelComRegistro</code> que adicionam funcionalidades a outras classes e implementavam tudo o que precisavam para seu funcionamento. Entretanto podem existir casos em que não seja possível implementar todas as funções na própria classe, deixando com que as classes que a estende implemente essas funções. Uma forma de fazer isso é través das <a href="https://docs.python.org/pt-br/3/library/abc.html">ABC</a> (<em>abstract base classes</em>, ou classes base abstratas).</p>
<h2>Sem uso de classes base abstratas</h2>
<p>Um exemplo de classe que não é possível implementar todas as funcionalidades foi dada no texto <a href="https://dev.to/acaverna/encapsulamento-da-logica-do-algoritmo-298e">Encapsulamento da lógica do algoritmo</a>, que discutia a leitura de valores do teclado até que um valor válido fosse lido (ou que repete a leitura caso um valor inválido tivesse sido informado). Nesse caso a classe <code>ValidaInput</code> implementava a lógica base de funcionamento, porém eram suas classes filhas (<code>ValidaNomeInput</code> e <code>ValidaNotaInput</code>) que implementavam as funções para tratar o que foi lido do teclado e verificar se é um valor válido ou não.</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ValidaInput</span><span class="p">:</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Valor inválido!'</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">valor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">valor</span>
<span class="k">class</span> <span class="nc">ValidaNomeInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nome inválido!'</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="n">entrada</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">title</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="n">valor</span> <span class="o">!=</span> <span class="s1">''</span>
<span class="k">class</span> <span class="nc">ValidaNotaInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nota inválida!'</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">entrada</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">valor</span> <span class="o"><=</span> <span class="mi">10</span>
</pre></div>
<p>Entretanto, esse código permite a criação de objetos da classe <code>ValidaInput</code> mesmo sem ter uma implementação das funções <code>transformar_entrada</code> e <code>validar_valor</code>. E a única mensagem de erro ocorreria ao tentar executar essas funções, o que poderia estar longe do problema real, que é a criação de um objeto a partir de uma classe que não prove todas as implementações das suas funções, o que seria semelhante a uma classe abstrata em outras linguagens.</p>
<div class="highlight"><pre><span></span><span class="n">obj</span> <span class="o">=</span> <span class="n">ValidaInput</span><span class="p">()</span>
<span class="c1"># Diversas linhas de código</span>
<span class="n">obj</span><span class="p">(</span><span class="s1">'Entrada: '</span><span class="p">)</span> <span class="c1"># Exceção NotImplementedError lançada</span>
</pre></div>
<h2>Com uso de classes base abstratas</h2>
<p>Seguindo a documentação da <a href="https://docs.python.org/pt-br/3/library/abc.html">ABC</a>, para utilizá-las é necessário informar a metaclasse <code>ABCMeta</code> na criação da classe, ou simplesmente estender a classe <code>ABC</code>, e decorar com <code>abstractmethod</code> as funções que as classes que a estenderem deverão implementar. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
<span class="k">class</span> <span class="nc">ValidaInput</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Valor inválido!'</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="o">...</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">valor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">valor</span>
</pre></div>
<p>Desta forma, ocorrerá um erro já ao tentar criar um objeto do tipo <code>ValidaInput</code>, dizendo quais são as funções que precisam ser implementadas. Porém funcionará normalmente ao criar objetos a partir das classes <code>ValidaNomeInput</code> e <code>ValidaNotaInput</code> visto que elas implementam essas funções.</p>
<div class="highlight"><pre><span></span><span class="n">obj</span> <span class="o">=</span> <span class="n">ValidaInput</span><span class="p">()</span> <span class="c1"># Exceção TypeError lançada</span>
<span class="n">nome_input</span> <span class="o">=</span> <span class="n">ValidaNomeInput</span><span class="p">()</span> <span class="c1"># Objeto criado</span>
<span class="n">nota_input</span> <span class="o">=</span> <span class="n">ValidaNotaInput</span><span class="p">()</span> <span class="c1"># Objeto criado</span>
</pre></div>
<p>Como essas funções não utilizam a referência ao objeto (<code>self</code>), ainda é possível decorar as funções com <code>staticmethod</code>, como:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
<span class="k">class</span> <span class="nc">ValidaInput</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Valor inválido!'</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="nd">@staticmethod</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="n">entrada</span><span class="p">):</span>
<span class="o">...</span>
<span class="nd">@staticmethod</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">valor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">valor</span>
<span class="k">class</span> <span class="nc">ValidaNomeInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nome inválido!'</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="n">entrada</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">title</span><span class="p">()</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="n">valor</span> <span class="o">!=</span> <span class="s1">''</span>
<span class="k">class</span> <span class="nc">ValidaNotaInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nota inválida!'</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">entrada</span><span class="p">)</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">valor</span> <span class="o"><=</span> <span class="mi">10</span>
</pre></div>
<p>Isso também seria válido para funções decoradas com <code>classmethod</code>, que receberiam a referência a classe (<code>cls</code>).</p>
<h2>Considerações</h2>
<p>Não é necessário utilizar ABC para fazer o exemplo discutido, porém ao utilizar essa biblioteca ficou mais explícito quais as funções que precisavam ser implementados nas classes filhas, ainda mais que sem utilizar ABC a classe base poderia nem ter as funções, com:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ValidaInput</span><span class="p">:</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Valor inválido!'</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">valor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">valor</span>
</pre></div>
<p>Como Python possui <a href="https://docs.python.org/pt-br/3/glossary.html#term-duck-typing">duck-typing</a>, não é necessário uma grande preocupação com os tipos, como definir e utilizar interfaces presentes em outras implementações de orientação a objetos, porém devido à herança múltipla, ABC pode ser utilizada como interface que não existe em Python, fazendo com que as classes implementem determinadas funções. Para mais a respeito desse assunto, recomendo as duas lives do dunossauro sobre ABC (<a href="https://www.youtube.com/watch?v=yLHV1__nZZw">1</a> e <a href="https://www.youtube.com/watch?v=erAXvsuihPQ">2</a>), e a apresentação do Luciano Ramalho sobre <a href="https://www.youtube.com/watch?v=AJK2LqrlnTE">type hints</a>.</p>
<p>Uma classe filha também não é obrigada a implementar todas as funções decoradas com <code>abstractmethod</code>, mas assim como a classe pai, não será possível criar objetos a partir dessa classe, apenas de uma classe filha dela que implemente as demais funções. Como se ao aplicar um <code>abstractmethod</code> tornasse a classe abstrata, e qualquer classe filha só deixasse de ser abstrata quando a última função decorada com <code>abstractmethod</code> for sobrescrita. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">abc</span> <span class="kn">import</span> <span class="n">ABC</span><span class="p">,</span> <span class="n">abstractmethod</span>
<span class="k">class</span> <span class="nc">A</span><span class="p">(</span><span class="n">ABC</span><span class="p">):</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">func1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="o">...</span>
<span class="nd">@abstractmethod</span>
<span class="k">def</span> <span class="nf">func2</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">B</span><span class="p">(</span><span class="n">A</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">func1</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'1'</span><span class="p">)</span>
<span class="k">class</span> <span class="nc">C</span><span class="p">(</span><span class="n">B</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">func2</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'2'</span><span class="p">)</span>
<span class="n">a</span> <span class="o">=</span> <span class="n">A</span><span class="p">()</span> <span class="c1"># Erro por não implementar func1 e func2</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">B</span><span class="p">()</span> <span class="c1"># Erro por não implementar func2</span>
<span class="n">c</span> <span class="o">=</span> <span class="n">C</span><span class="p">()</span> <span class="c1"># Objeto criado</span>
</pre></div>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Juros e lógicahttps://blog.nilo.pro.br/posts/2021-05-06-juros/2021-05-06T18:18:00+00:00<p>Um dos problemas de lógica de programação que mais quebram a cabeça de quem está começando são os problemas com cálculo de porcentagens ou juros.</p>
<p>Começa assim: Calcule 10% de aumento de um salário de R$2.500,00</p>
<p>Dependendo da base matemática do aluno, porcentagem se aprende na quarta/quinta série… alguns conceitos tem que ser relembrados. Como diz o nome, a 10% significa que a cada 100 do valor, você deve retirar 10. O cálculo é bem simples, pode ser realizado com uma multiplicação. Por exemplo: 10/100 * 2500, que resulta em 250. Até aí tudo bem, mas alguns alunos perguntam por que as vezes fazemos 0.1 * 2500. Bem, é apenas a forma de representar que muda, pois 10/100 é equivalente a 0.1, alias 100/1000 também e por aí vai. Mas está é a parte fácil e depois de acertar a notação, as coisas voltam a andar novamente.</p>
<p>Quando chegamos em repetições, problemas como cálculo da poupança começam a aparecer. Eles tem o seguinte formato: imagine que alguém deposita R$1.000,00 para poupança e que esta rende 1% ao mês (bons tempos). Depois de n meses, quanto a pessoa terá como saldo? O problema combina porcentagem com repetição. Vejamos como fica para 6 meeses:</p>
<div class="highlight"><pre><code class="language-python">saldo <span>=</span> <span>1000</span> <span># Valor inicial</span>
n <span>=</span> <span>6</span> <span># Número de meses</span>
juros <span>=</span> <span>0.01</span> <span># Juros mensal 1% = 1/100 = 0.01</span>
<span>for</span> mês <span>in</span> range(<span>1</span>, n <span>+</span> <span>1</span>):
saldo <span>*=</span> <span>1</span> <span>+</span> juros
<span>print</span>(f<span>"Mês ({mês}): {saldo:7.2f}"</span>)
</code></pre></div><p>que resulta em:</p>
<pre><code>Mês (1): 1010.00
Mês (2): 1020.10
Mês (3): 1030.30
Mês (4): 1040.60
Mês (5): 1051.01
Mês (6): 1061.52
</code></pre><p>Várias perguntas surgem.</p>
<h2 id="por-que-o-">Por que o <code>*=</code>?</h2>
<p>É que o juros é composto, ou seja, aplicado ao saldo precedente. Não se calcula 1% do saldo inicial e se multiplica pelo número de meses. Vai ficar mais claro depois com a fórmula.</p>
<h2 id="por-que-se-soma-1-ao-juros">Por que se soma 1 ao juros?</h2>
<p>Como estamos multiplicando o saldo, precisamos ajustar o novo saldo para que seja igual ao anterior mais o juros do mês. Em formato mais longo seria <code>saldo = saldo + juros * saldo</code>. Este cálculo pode ser simplificado, agrupando a variável saldo de forma que <code>saldo = saldo * (1 + juros)</code>. Como em Python podemos escrever <code>saldo = saldo *</code> como <code>saldo *=</code>, a expressão fica resumida a <code>saldo *= (1 + juros)</code>, veja que retirei os parênteses pois não são mais necessários.</p>
<h2 id="precisa-de-repetição-para-realizar-este-cálculo">Precisa de repetição para realizar este cálculo?</h2>
<p>Não, você pode ter uma solução analítica, aplicando a fórmula: `saldo = saldo * (1 + juros) ** mês).
Exemplo:</p>
<pre><code>>>> print(f"{1000 * (1 + 0.01) ** 6:7.2f}")
1061.52
</code></pre><h2 id="e-de-onde-vem-essa-exponenciação">E de onde vem essa exponenciação?</h2>
<p>Se voltarmos ao exemplo com repetição, veremos que:</p>
<p>No mês 1, o saldo é saldo * 1.01.</p>
<p>O saldo no mês 2 é saldo * 1.01 * 1.01</p>
<p>No mês 3, saldo * 1.01 * 1.01 * 1.01</p>
<p>E assim por diante, começamos a ver um padrão onde o 1.01 é multiplicado por ele mesmo o número de mês que estamos calculando. Podemos então escrever de forma genérica que <code>saldo *= 1.01 ** mês</code>, que é justamente a definição da exponenciação: <code>a ** n = a * a * a ... (n vezes)</code></p>
<h2 id="por-que-usamos-a-repetição">Por que usamos a repetição?</h2>
<p>Porque o curso é de lógica de programação e o professor quer que você tenha uma motivo para usar o for ou while.</p>
<h2 id="uma-questão-que-apareceu-hoje-no-stackoverflow-em-português">Uma questão que apareceu hoje no StackOverflow em Português</h2>
<p>A questão foi votada para baixo e logo depois fechada :-(</p>
<blockquote>
<p>Duas fabricantes de calçado disputam o mercado no Brasil. A empresa A tem produção de 10.000 pares/mês e um crescimento mensal de 15%. A empresa B, de 8.000 pares/mês e tem um crescimento mensal de 20%. Determinar o número de meses necessários para que a empresa B supere o número de pares produzidos pela empresa A.</p>
</blockquote>
<p>Vamos ver como ficaria isso na matemática:</p>
<p>Produção da empresa A: 10000 x 1.15<sup>m</sup></p>
<p>Produção da empresa B: 8000 x 1.20<sup>m</sup></p>
<p>Onde <em>m</em> é número de meses.</p>
<p>O que você procura é o valor de <em>m</em> quando:</p>
<p>8000 x 1.20<sup>m</sup> > 10000 x 1.15<sup>m</sup></p>
<p>Você pode resolver isso usando logaritmos, mas num curso de lógica de programação, o professor provavelmente espera que você varie o valor de m de 1 em 1.</p>
<p>Então, tente calcular a produção quando m = 1.
Compare a produção das duas fábricas (use as fórmulas acima). Se o valor de B não ultrapassar o de A, continue incrementando m de 1. Pare quando a produção de B for maior que A (a resposta é o valor de <em>m</em> quando isso acontecer).</p>
<div class="highlight"><pre><code class="language-python">m <span>=</span> <span>0</span>
<span>while</span> True:
prodA <span>=</span> <span>10000</span> <span>*</span> <span>1.15</span> <span>**</span> m
prodB <span>=</span> <span>8000</span> <span>*</span> <span>1.2</span> <span>**</span> m
<span>print</span>(f<span>"Prod A: {prodA:8.2f} Prod B: {prodB:8.2f} - mês: {m}"</span>)
<span>if</span> prodB <span>></span> prodA:
<span>break</span>
m <span>+=</span> <span>1</span>
<span>print</span>(f<span>"Meses para que a produção de B ultrapasse a produção de A: {m}"</span>)
</code></pre></div><p>que resulta em:</p>
<pre><code>Prod A: 10000.00 Prod B: 8000.00 - mês: 0
Prod A: 11500.00 Prod B: 9600.00 - mês: 1
Prod A: 13225.00 Prod B: 11520.00 - mês: 2
Prod A: 15208.75 Prod B: 13824.00 - mês: 3
Prod A: 17490.06 Prod B: 16588.80 - mês: 4
Prod A: 20113.57 Prod B: 19906.56 - mês: 5
Prod A: 23130.61 Prod B: 23887.87 - mês: 6
Meses para que a produção de B ultrapasse a produção de A: 6
</code></pre><p>E a solução analítica? Como falei o curso é de lógica de programação :-D</p>
<p>Mas você pode resolver a desigualdade usando logaritmos:</p>
<p>8000 x 1.20<sup>m</sup> > 10000 x 1.15<sup>m</sup></p>
<p>ln(8000) + m * ln(1.20) > ln(10000) + m * ln(1.15)</p>
<p>m * ln(1.20) - m * ln(1.15) > ln(10000) - ln(8000)</p>
<p>m * (ln(1.20) - ln(1.15)) > ln(10000) - ln(8000)</p>
<p>m = (ln(10000) - ln(8000))/(ln(1.20) - ln(1.15))</p>
<p>que em Python fica:</p>
<pre><code>from math import log
m = (log(10000) - log(8000))/(log(1.20) - log(1.15))
print(m)
</code></pre><p>que resulta em:</p>
<pre><code>5.243082071149164
</code></pre><p>Que podemos arredondar para 6, caso não consideremos frações de um mês.
Agora conhecemos o método analítico e o iterativo. Fica fácil de entender porque o segundo é mais usado em cursos de lógica de programação.</p>Nilo Ney Coutinho Menezeshttps://blog.nilo.pro.br/tags/python/Orientação a objetos de outra forma: Herança múltiplas e mixinstag:pythonclub.com.br,2021-05-03:/oo-de-outra-forma-4.html2021-05-03T18:00:00+00:00<p>No <a href="https://dev.to/acaverna/orientacao-a-objetos-de-outra-forma-heranca-3dm7">texto anterior</a> foi apresentando o conceito de herança, que herda toda a estrutura e comportamento de uma classe, podendo estendê-la com outros atributos e comportamentos. Esse texto apresentará a ideia de <a href="https://pt.wikipedia.org/wiki/Heran%C3%A7a_m%C3%BAltipla">herança múltipla</a>, e uma forma para se aproveitar esse recurso, através de mixins.</p>
<h2>Herança múltiplas</h2>
<p>Voltando ao sistema para lidar com dados das pessoas, onde algumas dessas pessoas possuem a possibilidade de acessar o sistema através de usuário e senha, também deseja-se permitir que outros sistemas autentiquem e tenham acesso os dados através de uma <a href="https://pt.wikipedia.org/wiki/Interface_de_programa%C3%A7%C3%A3o_de_aplica%C3%A7%C3%B5es">API</a>. Isso pode ser feito criando uma classe para representar os sistemas que terão permissão para acessar os dados. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Sistema</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">=</span> <span class="n">senha</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">==</span> <span class="n">usuario</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">==</span> <span class="n">senha</span>
</pre></div>
<p>Porém, esse código repete a implementação feita para <code>PessoaAutenticavel</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">PessoaAutenticavel</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">=</span> <span class="n">senha</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">==</span> <span class="n">usuario</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">==</span> <span class="n">senha</span>
</pre></div>
<p>Aproveitando que Python, diferente de outras linguagens, possui herança múltipla, é possível extrair essa lógica das classes, centralizando a implementação em uma outra classe e simplesmente herdá-la. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Autenticavel</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">=</span> <span class="n">senha</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">==</span> <span class="n">usuario</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">==</span> <span class="n">senha</span>
<span class="k">class</span> <span class="nc">PessoaAutenticavel</span><span class="p">(</span><span class="n">Autenticavel</span><span class="p">,</span> <span class="n">Pessoa</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">Sistema</span><span class="p">(</span><span class="n">Autenticavel</span><span class="p">):</span>
<span class="o">...</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">PessoaAutenticavel</span><span class="p">(</span><span class="n">nome</span><span class="o">=</span><span class="s1">'João'</span><span class="p">,</span> <span class="n">sobrenome</span><span class="o">=</span><span class="s1">'da Silva'</span><span class="p">,</span> <span class="n">idade</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span>
<span class="n">usuario</span><span class="o">=</span><span class="s1">'joao'</span><span class="p">,</span> <span class="n">senha</span><span class="o">=</span><span class="s1">'secreta'</span><span class="p">)</span>
</pre></div>
<p>A primeira coisa a ser observada são os argumentos <code>*args</code> e <code>**kwargs</code> no <code>__init__</code> da classe <code>Autenticavel</code>, eles são usados uma vez que não se sabe todos os argumentos que o <code>__init__</code> da classe que estenderá o <code>Autenticavel</code> espera receber, funcionando de forma dinâmica (mais sobre esse recurso pode ser visto na <a href="https://docs.python.org/pt-br/3/tutorial/controlflow.html#more-on-defining-functions">documentação do Python</a>).</p>
<p>A segunda coisa a ser verificada é que para a classe <code>PessoaAutenticavel</code>, agora cria em seus objetos, a estrutura tanto da classe <code>Pessoa</code>, quanto <code>Autenticavel</code>. Algo similar a versão sem orientação a objetos a baixo:</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa_autenticavel.py</span>
<span class="kn">import</span> <span class="nn">autenticavel</span>
<span class="kn">import</span> <span class="nn">pessoa</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="n">autenticavel</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">)</span>
</pre></div>
<p>Também vale observar que as classes <code>PessoaAutenticavel</code> e <code>Sistema</code> não precisam definir nenhuma função, uma vez que elas cumprem seus papéis apenas herdando outras classes, porém seria possível implementar funções específicas dessas classes, assim como sobrescrever as funções definidas por outras classes.</p>
<h2>Ordem de resolução de métodos</h2>
<p>Embora herança múltiplas sejam interessantes, existe um problema, se ambas as classes pai possuírem uma função com um mesmo nome, a classe filha deveria chamar qual das funções? A do primeiro pai? A do último? Para lidar com esse problema o Python usa o MRO (<em>method resolution order</em>, ordem de resolução do método), que consiste em uma tupla com a ordem de qual classe o Python usará para encontrar o método a ser chamado. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">PessoaAutenticavel</span><span class="o">.</span><span class="vm">__mro__</span><span class="p">)</span>
<span class="c1"># (<class '__main__.PessoaAutenticavel'>, <class '__main__.Autenticavel'>, <class '__main__.Pessoa'>, <class 'object'>)</span>
</pre></div>
<p>Por esse motivo que também foi possível chamar o <code>super().__init__</code> dentro de <code>Autenticavel</code>, que devido ao MRO, o Python chama o <code>__init__</code> da outra classe pai da classe que estendeu <code>Autenticavel</code>, em vez de precisar fazer um método <code>__init__</code> em <code>PessoaAutenticavel</code> chamando o <code>__init__</code> de todas as suas classes pais, como foi feito na versão sem orientação a objetos. E por isso a ordem <code>Autenticavel</code> e <code>Pessoa</code> na herança de <code>PessoaAutenticavel</code>, para fazer o MRO procurar os métodos primeiro em <code>Autenticavel</code> e depois em <code>Pessoa</code>.</p>
<p>Para tentar fugir da complexidade que pode ser herança múltipla, é possível escrever classes que tem por objetivo unicamente incluir alguma funcionalidade em outra, como o caso da classe <code>Autenticavel</code>, que pode ser herdada por qualquer outra classe do sistema para permitir o acesso ao sistema. Essas classes recebem o nome de mixins, e adiciona uma funcionalidade bem definida.</p>
<h2>Estendendo mixins</h2>
<p>Imagine se além de permitir o acesso ao sistema, também gostaríamos de registrar algumas tentativas de acesso, informando quando houve a tentativa e se o acesso foi concedido ou não. Como <code>Autenticavel</code> é uma classe, é possível extendê-la para implementar essa funcionalidade na função <code>autenticar</code>. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span>
<span class="k">class</span> <span class="nc">AutenticavelComRegistro</span><span class="p">(</span><span class="n">Autenticavel</span><span class="p">):</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">_get_data</span><span class="p">():</span>
<span class="k">return</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span><span class="o">.</span><span class="n">strftime</span><span class="p">(</span><span class="s1">'</span><span class="si">%d</span><span class="s1">/%m/%Y %T'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'{self._get_data()} Tentativa de acesso de </span><span class="si">{usuario}</span><span class="s1">'</span><span class="p">)</span>
<span class="n">acesso</span> <span class="o">=</span> <span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">)</span>
<span class="k">if</span> <span class="n">acesso</span><span class="p">:</span>
<span class="n">acesso_str</span> <span class="o">=</span> <span class="s1">'permitido'</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">acesso_str</span> <span class="o">=</span> <span class="s1">'negado'</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'{self._get_data()} Acesso de </span><span class="si">{usuario}</span><span class="s1"> </span><span class="si">{acesso_str}</span><span class="s1">'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">acesso</span>
<span class="k">class</span> <span class="nc">PessoaAutenticavelComRegistro</span><span class="p">(</span><span class="n">AutenticavelComRegistro</span><span class="p">,</span> <span class="n">Pessoa</span><span class="p">):</span>
<span class="o">...</span>
<span class="k">class</span> <span class="nc">SistemaAutenticavelComRegistro</span><span class="p">(</span><span class="n">AutenticavelComRegistro</span><span class="p">,</span> <span class="n">Sistema</span><span class="p">):</span>
<span class="o">...</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">PessoaAutenticavelComRegistro</span><span class="p">(</span>
<span class="n">nome</span><span class="o">=</span><span class="s1">'João'</span><span class="p">,</span> <span class="n">sobrenome</span><span class="o">=</span><span class="s1">'da Silva'</span><span class="p">,</span> <span class="n">idade</span><span class="o">=</span><span class="mi">20</span><span class="p">,</span>
<span class="n">usuario</span><span class="o">=</span><span class="s1">'joao'</span><span class="p">,</span> <span class="n">senha</span><span class="o">=</span><span class="s1">'secreta'</span><span class="p">,</span>
<span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">)</span>
<span class="c1"># Saída na tela:</span>
<span class="c1"># 23/04/2021 16:56:58 Tentativa de acesso de joao</span>
<span class="c1"># 23/04/2021 16:56:58 Acesso de joao permitido</span>
</pre></div>
<p>Essa implementação utiliza-se do <code>super()</code> para acessar a função <code>autenticar</code> da classe <code>Autenticavel</code> para não precisar reimplementar a autenticação. Porém, antes de chamá-la, manipula seus argumentos para registrar quem tentou acessar o sistema, assim como também manipula o seu retorno para registrar se o acesso foi permitido ou não.</p>
<p>Essa classe também permite analisar melhor a ordem em que as classes são consultadas quando uma função é chamada:</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">PessoaAutenticavelComRegistro</span><span class="o">.</span><span class="vm">__mro__</span><span class="p">)</span>
<span class="c1"># (<class '__main__.PessoaAutenticavelComRegistro'>, <class '__main__.AutenticavelComRegistro'>, <class '__main__.Autenticavel'>, <class '__main__.Pessoa'>, <class 'object'>)</span>
</pre></div>
<p>Que também pode ser visto na forma de um digrama de classes:</p>
<p><img alt="Diagrama de classes" src="http://pythonclub.com.br/feeds/images/eduardoklosowski/oo-de-outra-forma-4/mro.png" /></p>
<p>Onde é feito uma <a href="https://pt.wikipedia.org/wiki/Busca_em_profundidade">busca em profundidade</a>, como se a função fosse chamada no primeiro pai, e só se ela não for encontrada, busca-se no segundo pai e assim por diante. Também é possível observar a classe <code>object</code>, que sempre será a última classe, e é a classe pai de todas as outras classes do Python quando elas não possuirem um pai declarado explicitamente.</p>
<h2>Considerações</h2>
<p>Herança múltipla pode dificultar bastante o entendimento do código, principalmente para encontrar onde determinada função está definida, porém pode facilitar bastante o código. Um exemplo que usa bastante herança e mixins são as <em>views</em> baseadas em classe do django (<a href="https://docs.djangoproject.com/pt-br/3.2/topics/class-based-views/"><em>class-based views</em></a>), porém para facilitar a visualização existe o site <a href="https://ccbv.co.uk/">Classy Class-Based Views</a> que lista todas as classes, e os mixins utilizados em cada uma, como pode ser visto em "Ancestors" como na <a href="https://ccbv.co.uk/projects/Django/3.1/django.views.generic.edit/UpdateView/">UpdateView</a>, que é usado para criar uma página com formulário para editar um registro já existente no banco, assim ela usa mixins para pegar um objeto do banco (<code>SingleObjectMixin</code>), processar formulário baseado em uma tabela do banco (<code>ModelFormMixin</code>) e algumas outras funcionalidades necessárias para implementar essa página.</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Configurando limite de recursos em aplicações Java (JVM) no Kuberneteshttps://avelino.run/configurando-limite-de-recursos-em-aplicacoes-java-jvm-no-kubernetes/2021-05-01T00:00:00+00:00Fazer deploy de software desenvolvido usando tecnologias que foram criadas para ter escalabilidade vertical para escalar horizontalmente (micro serviço, nano serviço e etc) em produção pode gerar alguns desafios que não estamos preparados. Principalmente quando o software esta rodando em JVM e não foi declarado limites de recursos.
-Xms, -Xmx e seus problemas Ao estudar sobre a JVM você provavelmente passara pelos parâmetros de alocação inicial (Xms) e alocação máxima (Xmx) de memória, os parâmetros funcionam rigorosamente bem.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/gxzDHUoec_w" width="1" />Thiago Avelinohttps://avelino.run/blog/Falar sobre 'Assuntos Difíceis'https://avelino.run/falar-sobre-assuntos-dif%C3%ADceis/2021-04-29T00:00:00+00:00Em nossa vida é necessário encarar de frente assuntos considerados difíceis ou tabu com qualquer pessoa, para isso precisamos ter coragem e maturidade para lidar com naturalidade com qualquer tema - mesmo se ele nos tire da zona de conforto.
Para falar sobre esse assunto, vamos começar pelo porquê.
Por que é comum procrastinar uma conversa quando envolve assunto difícil?
Quando não falamos constantemente sobre um tipo de assunto ele se torna “difícil” por falta de familiaridade e sensação de desconforto.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/v0sxwO87Kd8" width="1" />Thiago Avelinohttps://avelino.run/blog/Iniciando com o ORM Pony no Pythonhttps://paulohrpinheiro.xyz/texts/python/2021-04-28-iniciando-com-o-orm-pony-no-python.html2021-04-28T00:00:00+00:00Depois de anos só no Django, estou eu sendo iniciado na simplicidade e elegância do Pony ORMBlog do PauloHRPinheirohttps://paulohrpinheiro.xyzOrientação a objetos de outra forma: Herançatag:pythonclub.com.br,2021-04-26:/oo-de-outra-forma-3.html2021-04-26T20:00:00+00:00<p>Algo que ajuda no desenvolvimento é a reutilização de código. Em orientação a objetos, essa reutilização pode ocorrer através de herança, onde um objeto pode se comportar como um objeto da sua própria classe, como também da classe que herdou.</p>
<h2>Adicionando funcionalidades</h2>
<p>Uma das utilidades da herança é estender uma classe para adicionar funcionalidades. Pensando no contexto das postagens anteriores, poderíamos querer criar um usuário e senha para algumas pessoas poderem acessar o sistema. Isso poderia ser feito adicionando atributos usuário e senha para as pessoas, além de uma função para validar se os dados estão corretos, e assim permitir o acesso ao sistema. Porém isso não pode ser feito para todas as pessoas, e sim apenas para aqueles que possuem permissão de acesso.</p>
<h3>Sem orientação a objetos</h3>
<p>Voltando a solução com dicionários (sem utilizar orientação a objetos), isso consistiria em criar um dicionário com a estrutura de uma pessoa, e em seguida estender essa estrutura com os novos campos de usuário e senha nesse mesmo dicionário, algo como:</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa.py</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'nome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">nome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'sobrenome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'idade'</span><span class="p">]</span> <span class="o">=</span> <span class="n">idade</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="n">pessoa</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{pessoa['nome']}</span><span class="s2"> </span><span class="si">{pessoa['sobrenome']}</span><span class="s2">"</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa_autenticavel.py</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'usuario'</span><span class="p">]</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'senha'</span><span class="p">]</span> <span class="o">=</span> <span class="n">senha</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="k">return</span> <span class="n">pessoa</span><span class="p">[</span><span class="s1">'usuario'</span><span class="p">]</span> <span class="o">==</span> <span class="n">usuario</span> <span class="ow">and</span> <span class="n">pessoa</span><span class="p">[</span><span class="s1">'senha'</span><span class="p">]</span> <span class="o">==</span> <span class="n">senha</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="kn">import</span> <span class="nn">pessoa_autenticavel</span>
<span class="n">p</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">pessoa_autenticavel</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">pessoa</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">(</span><span class="n">p</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">pessoa_autenticavel</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">))</span>
</pre></div>
<p>Porém nessa solução é possível que o programador esqueça de chamar as duas funções <code>init</code> diferentes, e como queremos que todo dicionário com a estrutura de <code>pessoa_autenticavel</code> contenha também a estrutura de <code>pessoa</code>, podemos chamar o <code>init</code> de pessoa dentro do <code>init</code> de <code>pessoa_autenticavel</code>:</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa_autenticavel.py</span>
<span class="kn">import</span> <span class="nn">pessoa</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="n">p</span><span class="p">[</span><span class="s1">'usuario'</span><span class="p">]</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="n">p</span><span class="p">[</span><span class="s1">'senha'</span><span class="p">]</span> <span class="o">=</span> <span class="n">senha</span>
<span class="o">...</span> <span class="c1"># Demais funções</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="kn">import</span> <span class="nn">pessoa_autenticavel</span>
<span class="n">p</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">pessoa_autenticavel</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">pessoa</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">(</span><span class="n">p</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">pessoa_autenticavel</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">))</span>
</pre></div>
<p>Nesse caso foi necessário alterar o nome do argumento <code>pessoa</code> da função <code>pessoa_autenticavel.init</code> para não conflitar com o outro módulo importado com esse mesmo nome. Porém ao chamar um <code>init</code> dentro de outro, temos a garantia de que o dicionário será compatível tanto com a estrutura pedida para ser criada pelo programador, quanto pelas estruturas pais dela.</p>
<h3>Com orientação a objetos</h3>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.nome}</span><span class="s1"> </span><span class="si">{self.sobrenome}</span><span class="s1">'</span>
<span class="k">class</span> <span class="nc">PessoaAutenticavel</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="n">Pessoa</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">=</span> <span class="n">senha</span>
<span class="k">def</span> <span class="nf">autenticar</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">==</span> <span class="n">usuario</span> <span class="ow">and</span> <span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">==</span> <span class="n">senha</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">PessoaAutenticavel</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Pessoa</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">(</span><span class="n">p</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">PessoaAutenticavel</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="n">p</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">))</span>
</pre></div>
<p>A principal novidade desse exemplo é que ao declarar a classe <code>PessoaAutenticavel</code> (filha), foi declarado a classe <code>Pessoa</code> (pai) entre parênteses, isso faz o interpretador Python criar uma cópia dessa classe estendendo-a com as novas funções que estamos criando. Porém pode ser um pouco redundante chamar <code>Pessoa.__init__</code> dentro da função <code>__init__</code> sendo que já foi declarado que ela estende <code>Pessoa</code>, podendo ser trocado por <code>super()</code>, que aponta para a classe que foi estendida. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">PessoaAutenticavel</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">,</span> <span class="n">usuario</span><span class="p">,</span> <span class="n">senha</span><span class="p">):</span>
<span class="nb">super</span><span class="p">()</span><span class="o">.</span><span class="fm">__init__</span><span class="p">(</span><span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">usuario</span> <span class="o">=</span> <span class="n">usuario</span>
<span class="bp">self</span><span class="o">.</span><span class="n">senha</span> <span class="o">=</span> <span class="n">senha</span>
<span class="o">...</span> <span class="c1"># Demais funções</span>
</pre></div>
<p>Assim se evita repetir o nome da classe, e já passa automaticamente a referência para <code>self</code>, assim como quando usamos o açúcar sintático apresentado na primeira postagem dessa série. E esse açúcar sintática também pode ser usado para chamar tanto as funções declaradas em <code>Pessoa</code> quanto em <code>PessoaAutenticavel</code>. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">p</span> <span class="o">=</span> <span class="n">PessoaAutenticavel</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">,</span> <span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">())</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p</span><span class="o">.</span><span class="n">autenticar</span><span class="p">(</span><span class="s1">'joao'</span><span class="p">,</span> <span class="s1">'secreta'</span><span class="p">))</span>
</pre></div>
<p>Esse método também facilita a utilização das funções, uma vez que não é necessário lembrar em qual classe que cada função foi declarada. Na verdade, como <code>PessoaAutenticavel</code> estende <code>Pessoa</code>, seria possível executar também <code>PessoaAutenticavel.nome_completo</code>, porém eles apontam para a mesma função.</p>
<h2>Sobrescrevendo uma função</h2>
<p>A classe <code>Pessoa</code> possui a função <code>nome_completo</code> que retorna uma <code>str</code> contento nome e sobrenome. Porém no Japão, assim como em outros países asiáticos, o sobrenome vem primeiro, e até <a href="https://noticias.uol.com.br/ultimas-noticias/efe/2019/06/24/japao-quer-voltar-a-ordem-tradicional-dos-nomes-abe-shinzo-nao-shinzo-abe.htm">estão pedindo para seguir a tradição deles ao falarem os nomes de japoneses</a>, como o caso do primeiro-ministro, mudando de Shinzo Abe para Abe Shinzo.</p>
<h3>Com orientação a objetos</h3>
<p>Isso também pode ser feito no sistema usando herança, porém em vez de criar uma nova função com outro nome, é possível criar uma função com o mesmo nome, sobrescrevendo a anterior, porém apenas para os objetos da classe filha. Algo semelhante ao que já foi feito com a função <code>__init__</code>. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Japones</span><span class="p">(</span><span class="n">Pessoa</span><span class="p">):</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.sobrenome}</span><span class="s1"> </span><span class="si">{self.nome}</span><span class="s1">'</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">p2</span> <span class="o">=</span> <span class="n">Japones</span><span class="p">(</span><span class="s1">'Shinzo'</span><span class="p">,</span> <span class="s1">'Abe'</span><span class="p">,</span> <span class="mi">66</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">())</span> <span class="c1"># João da Silva</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p2</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">())</span> <span class="c1"># Abe Shinzo</span>
</pre></div>
<p>Essa relação de herança traz algo interessante, todo objeto da classe <code>Japones</code> se comporta como um objeto da classe <code>Pessoa</code>, porém a relação inversa não é verdade. Assim como podemos dizer que todo japonês é uma pessoa, mas nem todas as pessoas são japonesas. Ser japonês é um caso mais específico de pessoa, assim como as demais nacionalidades.</p>
<h3>Sem orientação a objetos</h3>
<p>Esse comportamento de sobrescrever a função <code>nome_completo</code> não é tão simples de replicar em uma estrutura de dicionário, porém é possível fazer. Porém como uma pessoa pode ser tanto japonês quanto não ser, não é possível saber de antemão para escrever no código <code>pessoa.nome_completo</code> ou <code>japones.nome_completo</code>, que diferente do exemplo da autenticação, agora são duas funções diferentes, isso precisa ser descoberto dinamicamente quando se precisar chamar a função.</p>
<p>Uma forma de fazer isso é guardar uma referência para a função que deve ser chamada dentro da própria estrutura. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa.py</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'nome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">nome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'sobrenome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'idade'</span><span class="p">]</span> <span class="o">=</span> <span class="n">idade</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'nome_completo'</span><span class="p">]</span> <span class="o">=</span> <span class="n">nome_completo</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="n">pessoa</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{pessoa['nome']}</span><span class="s2"> </span><span class="si">{pessoa['sobrenome']}</span><span class="s2">"</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: japones.py</span>
<span class="kn">import</span> <span class="nn">pessoa</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">japones</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">(</span><span class="n">japones</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="n">japones</span><span class="p">[</span><span class="s1">'nome_completo'</span><span class="p">]</span> <span class="o">=</span> <span class="n">nome_completo</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="n">japones</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{pessoa['sobrenome']}</span><span class="s2"> </span><span class="si">{pessoa['nome']}</span><span class="s2">"</span>
</pre></div>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="kn">import</span> <span class="nn">japones</span>
<span class="n">p1</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p1</span><span class="p">,</span> <span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">p2</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">japones</span><span class="o">.</span><span class="n">init</span><span class="p">(</span><span class="n">p2</span><span class="p">,</span> <span class="s1">'Shinzo'</span><span class="p">,</span> <span class="s1">'Abe'</span><span class="p">,</span> <span class="mi">66</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="p">[</span><span class="s1">'nome_completo'</span><span class="p">](</span><span class="n">p1</span><span class="p">))</span> <span class="c1"># João da Silva</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p2</span><span class="p">[</span><span class="s1">'nome_completo'</span><span class="p">](</span><span class="n">p2</span><span class="p">))</span> <span class="c1"># Abe Shinzo</span>
</pre></div>
<p>Perceba que a forma de chamar a função foi alterada. O que acontece na prática é que toda função que pode ser sobrescrita não é chamada diretamente, e sim a partir de uma referência, e isso gera um custo computacional adicional. Como esse custo não é tão alto (muitas vezes sendo quase irrelevante), esse é o comportamento adotado em várias linguagens, porém em C++, por exemplo, existe a palavra-chave <code>virtual</code> para descrever quando uma função pode ser sobrescrita ou não.</p>
<h2>Considerações</h2>
<p>Herança é um mecanismo interessante para ser explorado com o objetivo de reaproveitar código e evitar repeti-lo. Porém isso pode vir com alguns custos, seja computacional durante sua execução, seja durante a leitura do código, sendo necessário verificar diversas classes para saber o que de fato está sendo executado, porém isso também pode ser usado para ocultar e abstrair lógicas mais complicadas, como eu já comentei em outra <a href="https://dev.to/acaverna/encapsulamento-da-logica-do-algoritmo-298e">postagem</a>.</p>
<p>Herança também permite trabalhar com generalização e especialização, podendo descrever o comportamento mais geral, ou mais específico. Ou simplesmente só adicionar mais funcionalidades a uma classe já existente.</p>
<p>Assim como foi utilizado o <code>super()</code> para chamar a função <code>__init__</code> da classe pai, é possível utilizá-lo para chamar qualquer outra função. Isso permite, por exemplo, tratar os argumentos da função, aplicando modificações antes de chamar a função original, ou seu retorno, executando algum processamento em cima do retorno dela, não precisando rescrever toda a função.</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Orientação a objetos de outra forma: Métodos estáticos e de classestag:pythonclub.com.br,2021-04-19:/oo-de-outra-forma-2.html2021-04-19T20:00:00+00:00<p>Na <a href="https://dev.to/acaverna/orientacao-a-objetos-de-outra-forma-classes-e-objetos-3mfd">postagem anterior</a> foi apresentado o <code>self</code>, nessa postagem será discutido mais a respeito desse argumento, considerando opções para ele e suas aplicações.</p>
<h2>Métodos estáticos</h2>
<p>Nem todas as funções de uma classe precisam receber uma referência de um objeto para lê-lo ou alterá-lo, muitas vezes uma função pode fazer o seu papel apenas com os dados passados como argumento, por exemplo, receber um nome e validar se ele possui pelo menos três caracteres sem espaço. Dessa forma, essa função poderia ser colocada fora do escopo da classe, porém para facilitar sua chamada, e possíveis alterações (que será discutido em outra postagem), é possível colocar essa função dentro da classe e informar que ela não receberá o argumento <code>self</code> com o decorador <code>@staticmethod</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="o">...</span> <span class="c1"># Demais funções</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">valida_nome</span><span class="p">(</span><span class="n">nome</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">nome</span><span class="p">)</span> <span class="o">>=</span> <span class="mi">3</span> <span class="ow">and</span> <span class="s1">' '</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">nome</span>
</pre></div>
<p>Dessa forma, essa função pode ser chamada diretamente de um objeto pessoa, ou até mesmo diretamente da classe, sem precisar criar um objeto primeiro:</p>
<div class="highlight"><pre><span></span><span class="c1"># Chamando diretamente da classe</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Pessoa</span><span class="o">.</span><span class="n">valida_nome</span><span class="p">(</span><span class="s1">'João'</span><span class="p">))</span>
<span class="c1"># Chamando através de um objeto do tipo Pessoa</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">valida_nome</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">nome</span><span class="p">))</span>
</pre></div>
<p>E essa função também pode ser utilizada dendro de outras funções, como validar o nome na criação de uma pessoa, de forma que caso o nome informado seja válido, será criado um objeto do tipo Pessoa, e caso o nome seja inválido, será lançado uma exceção:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="k">if</span> <span class="ow">not</span> <span class="bp">self</span><span class="o">.</span><span class="n">valida_nome</span><span class="p">(</span><span class="n">nome</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">ValueError</span><span class="p">(</span><span class="s1">'Nome inválido'</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="o">...</span> <span class="c1"># Demais funções</span>
<span class="nd">@staticmethod</span>
<span class="k">def</span> <span class="nf">valida_nome</span><span class="p">(</span><span class="n">nome</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">len</span><span class="p">(</span><span class="n">nome</span><span class="p">)</span> <span class="o">>=</span> <span class="mi">3</span> <span class="ow">and</span> <span class="s1">' '</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">nome</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span> <span class="c1"># Cria objeto</span>
<span class="n">p2</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'a'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span> <span class="c1"># Lança ValueError: Nome inválido</span>
</pre></div>
<h2>Métodos da classe</h2>
<p>Entretanto algumas funções podem precisar de um meio termo, necessitar acessar o contexto da classe, porém sem necessitar de um objeto. Isso é feito através do decorador <code>@classmethod</code>, onde a função decorada com ele, em vez de receber um objeto como primeiro argumento, recebe a própria classe.</p>
<p>Para demonstrar essa funcionalidade será implementado um <em>id</em> auto incremental para os objetos da classe <code>Pessoa</code>:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="n">total_de_pessoas</span> <span class="o">=</span> <span class="mi">0</span>
<span class="nd">@classmethod</span>
<span class="k">def</span> <span class="nf">novo_id</span><span class="p">(</span><span class="bp">cls</span><span class="p">):</span>
<span class="bp">cls</span><span class="o">.</span><span class="n">total_de_pessoas</span> <span class="o">+=</span> <span class="mi">1</span>
<span class="k">return</span> <span class="bp">cls</span><span class="o">.</span><span class="n">total_de_pessoas</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">id</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">novo_id</span><span class="p">()</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">id</span><span class="p">)</span> <span class="c1"># Imprime 1</span>
<span class="n">p2</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'Maria'</span><span class="p">,</span> <span class="s1">'dos Santos'</span><span class="p">,</span> <span class="mi">18</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p2</span><span class="o">.</span><span class="n">id</span><span class="p">)</span> <span class="c1"># Imprime 2</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Pessoa</span><span class="o">.</span><span class="n">total_de_pessoas</span><span class="p">)</span> <span class="c1"># Imprime 2</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">total_de_pessoas</span><span class="p">)</span> <span class="c1"># Imprime 2</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p2</span><span class="o">.</span><span class="n">total_de_pessoas</span><span class="p">)</span> <span class="c1"># Imprime 2</span>
</pre></div>
<p>Nesse código é criado uma variável <code>total_de_pessoas</code> dentro do escopo da classe <code>Pessoas</code>, e que é compartilhado tanto pela classe, como pelos objetos dessa classe, diferente de declará-la com <code>self.</code> dentro do <code>__init__</code>, onde esse valor pertenceria apenas ao objeto, e não é compartilhado com os demais objetos. Declarar variáveis dentro do contexto da classe é similar ao se declarar variáveis com <code>static</code> em outras linguagens, assim como o <code>@classmethod</code> é semelhante a declaração de funções com <code>static</code>.</p>
<p>As funções declaradas com <code>@classmethod</code> também podem ser chamadas sem a necessidade de se criar um objeto, como <code>Pessoa.novo_id()</code>, embora que para essa função específica isso não faça muito sentido, ou receber outros argumentos, tudo depende do que essa função fará.</p>
<h2>Considerações</h2>
<p>Embora possa parecer confuso identificar a diferença de uma função de um objeto (função sem decorador), função de uma classe (com decorador <code>@classmethod</code>) e função sem acesso a nenhum outro contexto (com decorador <code>@staticmethod</code>), essa diferença fica mais clara ao se analisar o primeiro argumento recebido por cada tipo de função. Podendo ser a referência a um objeto (<code>self</code>) e assim necessitando que um objeto seja criado anteriormente, ser uma classe (<code>cls</code>) e não necessitando receber um objeto, ou simplesmente não recebendo nenhum argumento especial, apenas os demais argumentos necessários para a função. Sendo diferenciados pelo uso dos decoradores.</p>
<p>Na orientação a objetos implementada pelo Python, algumas coisas podem ficar confusas quando se mistura com nomenclaturas de outras linguagens que possuem implementações diferentes. A linguagem Java, por exemplo, utiliza a palavra-chave <code>static</code> para definir os atributos e métodos de classe, enquanto no Python um método estático é aquele que não acessa nem um objeto, nem uma classe, devendo ser utilizado o escopo da classe e o decorador <code>@classmethod</code> para se criar atributos e métodos da classe.</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Orientação a objetos de outra forma: Classes e objetostag:pythonclub.com.br,2021-04-12:/oo-de-outra-forma-1.html2021-04-12T18:00:00+00:00<p>Nas poucas e raríssimas lives que eu fiz na <a href="https://www.twitch.tv/eduardoklosowski">Twitch</a>, surgiu a ideia de escrever sobre programação orientada a objetos em <a href="https://www.python.org/">Python</a>, principalmente por algumas diferenças de como ela foi implementada nessa linguagem. Aproveitando o tema, vou fazer uma série de postagens dando uma visão diferente sobre orientação a objetos. E nessa primeira postagem falarei sobre classes e objetos.</p>
<h2>Usando um dicionário</h2>
<p>Entretanto, antes de começar com orientação a objetos, gostaria de apresentar e discutir alguns exemplos sem utilizar esse paradigma de programação.</p>
<p>Pensando em um sistema que precise manipular dados de pessoas, é possível utilizar os <a href="https://docs.python.org/pt-br/3/library/stdtypes.html#mapping-types-dict">dicionários</a> do Python para agrupar os dados de uma pessoa em uma única variável, como no exemplo a baixo:</p>
<div class="highlight"><pre><span></span><span class="n">pessoa</span> <span class="o">=</span> <span class="p">{</span>
<span class="s1">'nome'</span><span class="p">:</span> <span class="s1">'João'</span><span class="p">,</span>
<span class="s1">'sobrenome'</span><span class="p">:</span> <span class="s1">'da Silva'</span><span class="p">,</span>
<span class="s1">'idade'</span><span class="p">:</span> <span class="mi">20</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<p>Onde os dados poderiam ser acessados através da variável e do nome do dado desejado, como:</p>
<div class="highlight"><pre><span></span><span class="nb">print</span><span class="p">(</span><span class="n">pessoa</span><span class="p">[</span><span class="s1">'nome'</span><span class="p">])</span> <span class="c1"># Imprimindo João</span>
</pre></div>
<p>Assim, todos os dados de uma pessoa ficam agrupados em uma variável, o que facilita bastante a programação, visto que não é necessário criar uma variável para cada dado, e quando se manipula os dados de diferentes pessoas fica muito mais fácil identificar de qual pessoa aquele dado se refere, bastando utilizar variáveis diferentes.</p>
<h3>Função para criar o dicionário</h3>
<p>Apesar de prático, é necessário replicar essa estrutura de dicionário toda vez que se desejar utilizar os dados de uma nova pessoa. Para evitar a repetição de código, a criação desse dicionário pode ser feita dentro de uma função que pode ser colocada em um módulo <code>pessoa</code> (arquivo, nesse caso com o nome de <code>pessoa.py</code>):</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa.py</span>
<span class="k">def</span> <span class="nf">nova</span><span class="p">(</span><span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="k">return</span> <span class="p">{</span>
<span class="s1">'nome'</span><span class="p">:</span> <span class="n">nome</span><span class="p">,</span>
<span class="s1">'sobrenome'</span><span class="p">:</span> <span class="n">sobrenome</span><span class="p">,</span>
<span class="s1">'idade'</span><span class="p">:</span> <span class="n">idade</span><span class="p">,</span>
<span class="p">}</span>
</pre></div>
<p>E para criar o dicionário que representa uma pessoa, basta importar esse módulo (arquivo) e chamar a função <code>nova</code>:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">pessoa</span><span class="o">.</span><span class="n">nova</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">p2</span> <span class="o">=</span> <span class="n">pessoa</span><span class="o">.</span><span class="n">nova</span><span class="p">(</span><span class="s1">'Maria'</span><span class="p">,</span> <span class="s1">'dos Santos'</span><span class="p">,</span> <span class="mi">18</span><span class="p">)</span>
</pre></div>
<p>Desta forma, garante-se que todos os dicionários representando pessoas terão os campos desejados e devidamente preenchidos.</p>
<h3>Função com o dicionário</h3>
<p>Também é possível criar algumas funções para executar operações com os dados desses dicionários, como pegar o nome completo da pessoa, trocar o seu sobrenome, ou fazer aniversário (o que aumentaria a idade da pessoa em um ano):</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa.py</span>
<span class="k">def</span> <span class="nf">nova</span><span class="p">(</span><span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="o">...</span> <span class="c1"># Código abreviado</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="n">pessoa</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s2">"</span><span class="si">{pessoa['nome']}</span><span class="s2"> </span><span class="si">{pessoa['sobrenome']}</span><span class="s2">"</span>
<span class="k">def</span> <span class="nf">trocar_sobrenome</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'sobrenome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="k">def</span> <span class="nf">fazer_aniversario</span><span class="p">(</span><span class="n">pessoa</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'idade'</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
<p>E sendo usado como:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">pessoa</span><span class="o">.</span><span class="n">nova</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">trocar_sobrenome</span><span class="p">(</span><span class="n">p1</span><span class="p">,</span> <span class="s1">'dos Santos'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">pessoa</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">(</span><span class="n">p1</span><span class="p">))</span>
<span class="n">pessoa</span><span class="o">.</span><span class="n">fazer_aniversario</span><span class="p">(</span><span class="n">p1</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="p">[</span><span class="s1">'idade'</span><span class="p">])</span>
</pre></div>
<p>Nesse caso, pode-se observar que todas as funções aqui implementadas seguem o padrão de receber o dicionário que representa a pessoa como primeiro argumento, podendo ter outros argumentos ou não conforme a necessidade, acessando e alterando os valores desse dicionário.</p>
<h2>Versão com orientação a objetos</h2>
<p>Antes de entrar na versão orientada a objetos propriamente dita dos exemplos anteriores, vou fazer uma pequena alteração para facilitar o entendimento posterior. A função <code>nova</code> será separada em duas partes, a primeira que criará um dicionário, e chamará uma segunda função (<code>init</code>), que receberá esse dicionário como primeiro argumento (seguindo o padrão das demais funções) e criará sua estrutura com os devidos valores.</p>
<div class="highlight"><pre><span></span><span class="c1"># Arquivo: pessoa.py</span>
<span class="k">def</span> <span class="nf">init</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'nome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">nome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'sobrenome'</span><span class="p">]</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="n">pessoa</span><span class="p">[</span><span class="s1">'idade'</span><span class="p">]</span> <span class="o">=</span> <span class="n">idade</span>
<span class="k">def</span> <span class="nf">nova</span><span class="p">(</span><span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="n">pessoa</span> <span class="o">=</span> <span class="p">{}</span>
<span class="n">init</span><span class="p">(</span><span class="n">pessoa</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">)</span>
<span class="k">return</span> <span class="n">pessoa</span>
<span class="o">...</span> <span class="c1"># Demais funções do arquivo</span>
</pre></div>
<p>Porém isso não muda a forma de uso:</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">pessoa</span>
<span class="n">p1</span> <span class="o">=</span> <span class="n">pessoa</span><span class="o">.</span><span class="n">nova</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
</pre></div>
<h3>Função para criar uma pessoa</h3>
<p>A maioria das linguagens de programação que possuem o paradigma de programação orientado a objetos faz o uso de classes para definir a estrutura dos objetos. O Python também utiliza classes, que podem ser definidas com a palavra-chave <code>class</code> seguidas de um nome para ela. E dentro dessa estrutura, podem ser definidas funções para manipular os objetos daquela classe, que em algumas linguagens também são chamadas de métodos (funções declaradas dentro do escopo uma classe).</p>
<p>Para converter o dicionário para uma classe, o primeiro passo é implementar uma função para criar a estrutura desejada. Essa função deve possui o nome <code>__init__</code>, e é bastante similar a função <code>init</code> do código anterior:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
</pre></div>
<p>As diferenças são que agora o primeiro parâmetro se chama <code>self</code>, que é um padrão utilizado no Python, e em vez de usar colchetes e aspas para acessar os dados, aqui basta utilizar o ponto e o nome do dado desejado (que aqui também pode ser chamado de atributo, visto que é uma variável do objeto). A função <code>nova</code> implementada anteriormente não é necessária, a própria linguagem cria um objeto e passa ele como primeiro argumento para o <code>__init__</code>. E assim para se criar um objeto da classe <code>Pessoa</code> basta chamar a classe como se fosse uma função, ignorando o argumento <code>self</code> e informando os demais, como se estivesse chamando a função <code>__init__</code> diretamente:</p>
<div class="highlight"><pre><span></span><span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'da Silva'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
</pre></div>
<p>Nesse caso, como a própria classe cria um contexto diferente para as funções (escopo ou <em>namespace</em>), não está mais sendo utilizado arquivos diferentes, porém ainda é possível fazê-lo, sendo necessário apenas fazer o <code>import</code> adequado. Mas para simplificação, tanto a declaração da classe, como a criação do objeto da classe <code>Pessoa</code> podem ser feitas no mesmo arquivo, assim como os demais exemplos dessa postagem.</p>
<h3>Outras funções</h3>
<p>As demais funções feitas anteriormente para o dicionário também podem ser feitas na classe <code>Pessoa</code>, seguindo as mesmas diferenças já apontadas anteriormente:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">Pessoa</span><span class="p">:</span>
<span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nome</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">,</span> <span class="n">idade</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">nome</span> <span class="o">=</span> <span class="n">nome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">=</span> <span class="n">idade</span>
<span class="k">def</span> <span class="nf">nome_completo</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="k">return</span> <span class="sa">f</span><span class="s1">'</span><span class="si">{self.nome}</span><span class="s1"> </span><span class="si">{self.sobrenome}</span><span class="s1">'</span>
<span class="k">def</span> <span class="nf">trocar_sobrenome</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">sobrenome</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">sobrenome</span> <span class="o">=</span> <span class="n">sobrenome</span>
<span class="k">def</span> <span class="nf">fazer_aniversario</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">idade</span> <span class="o">+=</span> <span class="mi">1</span>
</pre></div>
<p>Para se chamar essas funções, basta acessá-las através do contexto da classe, passando o objeto criado anteriormente como primeiro argumento:</p>
<div class="highlight"><pre><span></span><span class="n">p1</span> <span class="o">=</span> <span class="n">Pessoa</span><span class="p">(</span><span class="s1">'João'</span><span class="p">,</span> <span class="s1">'dos Santos'</span><span class="p">,</span> <span class="mi">20</span><span class="p">)</span>
<span class="n">Pessoa</span><span class="o">.</span><span class="n">trocar_sobrenome</span><span class="p">(</span><span class="n">p1</span><span class="p">,</span> <span class="s1">'dos Santos'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">Pessoa</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">(</span><span class="n">p1</span><span class="p">))</span>
<span class="n">Pessoa</span><span class="o">.</span><span class="n">fazer_aniversario</span><span class="p">(</span><span class="n">p1</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">idade</span><span class="p">)</span>
</pre></div>
<p>Essa sintaxe é bastante semelhante a versão sem orientação a objetos implementada anteriormente. Porém quando se está utilizando objetos, é possível chamar essas funções com uma outra sintaxe, informando primeiro o objeto, seguido de ponto e o nome da função desejada, com a diferença de que não é mais necessário informar o objeto como primeiro argumento. Como a função foi chamada através de um objeto, o próprio Python se encarrega de passá-lo para o argumento <code>self</code>, sendo necessário informar apenas os demais argumentos:</p>
<div class="highlight"><pre><span></span><span class="n">p1</span><span class="o">.</span><span class="n">trocar_sobrenome</span><span class="p">(</span><span class="s1">'dos Santos'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">nome_completo</span><span class="p">())</span>
<span class="n">p1</span><span class="o">.</span><span class="n">fazer_aniversario</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">p1</span><span class="o">.</span><span class="n">idade</span><span class="p">)</span>
</pre></div>
<p>Existem algumas diferenças entre as duas sintaxes, porém isso será tratado posteriormente. Por enquanto a segunda sintaxe pode ser vista como um <a href="https://pt.wikipedia.org/wiki/A%C3%A7%C3%BAcar_sint%C3%A1tico">açúcar sintático</a> da primeira, ou seja, uma forma mais rápida e fácil de fazer a mesma coisa que a primeira, e por isso sendo a recomendada.</p>
<h2>Considerações</h2>
<p>Como visto nos exemplos, programação orientada a objetos é uma técnica para juntar variáveis em uma mesma estrutura e facilitar a escrita de funções que seguem um determinado padrão, recebendo a estrutura como argumento, porém a sintaxe mais utilizada no Python para chamar as funções de um objeto (métodos) posiciona a variável que guarda a estrutura antes do nome da função, em vez do primeiro argumento.</p>
<p>No Python, o argumento da estrutura ou objeto (<code>self</code>) aparece explicitamente como primeiro argumento da função, enquanto em outras linguagens essa variável pode receber outro nome (como <code>this</code>) e não aparece explicitamente nos argumentos da função, embora essa variável tenha que ser criada dentro do contexto da função para permitir manipular o objeto.</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Funções in place ou cópia de valortag:pythonclub.com.br,2021-03-29:/funcao-inplace-ou-copia-de-valor.html2021-03-29T15:00:00+00:00<p>Eventualmente observo dificuldades de algumas pessoas em usar corretamente alguma função, seja porque a função deveria ser executada isoladamente, e utilizado a própria variável que foi passada como argumento posteriormente, seja porque deveria se atribuir o retorno da função a alguma variável, e utilizar essa nova variável. No Python, essa diferença pode ser observada nos métodos das listas <code>sort</code> e <code>reverse</code> para as funções <code>sorted</code> e <code>reversed</code>, que são implementadas com padrões diferentes, <em>in place</em> e cópia de valor respectivamente. Assim pretendo discutir esses dois padrões de funções, comentando qual a diferença e o melhor caso de aplicação de cada padrão.</p>
<h2>Função de exemplo</h2>
<p>Para demonstrar como esses padrões funcionam, será implementado uma função que recebe uma lista e calcula o dobro dos valores dessa lista. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">entrada</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="c1"># Execução da função</span>
<span class="n">resultado</span> <span class="o">=</span> <span class="p">[</span><span class="mi">10</span><span class="p">,</span> <span class="mi">4</span><span class="p">,</span> <span class="mi">16</span><span class="p">,</span> <span class="mi">12</span><span class="p">,</span> <span class="mi">8</span><span class="p">]</span>
</pre></div>
<h3>Função com in place</h3>
<p>A ideia do padrão <em>in place</em> é alterar a própria variável recebida pela função (ou o próprio objeto, caso esteja lidando com orientação a objetos). Neste caso, bastaria calcular o dobro do valor de cada posição da lista, e sobrescrever a posição com seu resultado. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span>
<span class="k">def</span> <span class="nf">dobro_inplace</span><span class="p">(</span><span class="n">lista</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-></span> <span class="kc">None</span><span class="p">:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">lista</span><span class="p">)):</span>
<span class="n">lista</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">lista</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">retorno</span> <span class="o">=</span> <span class="n">dobro_inplace</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: valores | Tipo: {type(valores)} | Valor: </span><span class="si">{valores}</span><span class="s1">'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: retorno | Tipo: {type(retorno)} | Valor: </span><span class="si">{retorno}</span><span class="s1">'</span><span class="p">)</span>
</pre></div>
<p>Resultado da execução:</p>
<div class="highlight"><pre><span></span><span class="err">Variável: valores | Tipo: <class 'list'> | Valor: [10, 4, 16, 12, 8]</span>
<span class="err">Variável: retorno | Tipo: <class 'NoneType'> | Valor: None</span>
</pre></div>
<p>Com essa execução é possível observar que os valores da lista foram alterados, e que o retorno da função é nulo (<code>None</code>), ou seja, a função alterou a própria lista passada como argumento. Outro ponto importante a ser observado é a assinatura da função (tipo dos argumentos e do retorno da função), que recebe uma lista de inteiros e não tem retorno ou é nulo (<code>None</code>). Dessa forma embora seja possível chamar essa função diretamente quando está se informando os argumentos de outra função, como <code>print(dobro_inplace(valores))</code>, a função <code>print</code> receberia <code>None</code> e não a lista como argumento.</p>
<h3>Função com cópia de valor</h3>
<p>A ideia do padrão cópia de valor é criar uma cópia do valor passado como argumento e retornar essa cópia, sem alterar a variável recebida (ou criando um novo objeto, no caso de orientação a objetos). Neste caso, é necessário criar uma nova lista e adicionar nela os valores calculados. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span>
<span class="k">def</span> <span class="nf">dobro_copia</span><span class="p">(</span><span class="n">lista</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
<span class="n">nova_lista</span> <span class="o">=</span> <span class="p">[]</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">lista</span><span class="p">)):</span>
<span class="n">nova_lista</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="mi">2</span> <span class="o">*</span> <span class="n">lista</span><span class="p">[</span><span class="n">i</span><span class="p">])</span>
<span class="k">return</span> <span class="n">nova_lista</span>
<span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">retorno</span> <span class="o">=</span> <span class="n">dobro_copia</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: valores | Tipo: {type(valores)} | Valor: </span><span class="si">{valores}</span><span class="s1">'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: retorno | Tipo: {type(retorno)} | Valor: </span><span class="si">{retorno}</span><span class="s1">'</span><span class="p">)</span>
</pre></div>
<p>Resultado da execução:</p>
<div class="highlight"><pre><span></span><span class="err">Variável: valores | Tipo: <class 'list'> | Valor: [5, 2, 8, 6, 4]</span>
<span class="err">Variável: retorno | Tipo: <class 'list'> | Valor: [10, 4, 16, 12, 8]</span>
</pre></div>
<p>Com essa execução é possível observar que a variável <code>valores</code> continua com os valores que tinha antes da execução da função, e a variável retorno apresenta uma lista com os dobros, ou seja, a função não altera a lista passada como argumento e retorna uma nova lista com os valores calculados. Observado a assinatura da função, ela recebe uma lista de inteiros e retorna uma lista de inteiros. Isso permite chamar essa função diretamente nos argumentos para outra função, como <code>print(dobro_copia(valores))</code>, nesse caso a função <code>print</code> receberia a lista de dobros como argumento. Porém caso o retorno da função não seja armazenado, parecerá que a função não fez nada, ou não funcionou. Então em alguns casos, quando o valor anterior não é mais necessário, pode-se reatribuir o retorno da função a própria variável passada como argumento:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="n">dobro_copia</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
</pre></div>
<h3>Função híbrida</h3>
<p>Ainda é possível mesclar os dois padrões de função, alterando o valor passado e retornando-o. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">typing</span> <span class="kn">import</span> <span class="n">List</span>
<span class="k">def</span> <span class="nf">dobro_hibrido</span><span class="p">(</span><span class="n">lista</span><span class="p">:</span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">])</span> <span class="o">-></span> <span class="n">List</span><span class="p">[</span><span class="nb">int</span><span class="p">]:</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">lista</span><span class="p">)):</span>
<span class="n">lista</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">=</span> <span class="mi">2</span> <span class="o">*</span> <span class="n">lista</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
<span class="k">return</span> <span class="n">lista</span>
<span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">retorno</span> <span class="o">=</span> <span class="n">dobro_hibrido</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: valores | Tipo: {type(valores)} | Valor: </span><span class="si">{valores}</span><span class="s1">'</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">'Variável: retorno | Tipo: {type(retorno)} | Valor: </span><span class="si">{retorno}</span><span class="s1">'</span><span class="p">)</span>
</pre></div>
<p>Resultado da execução:</p>
<div class="highlight"><pre><span></span><span class="err">Variável: valores | Tipo: <class 'list'> | Valor: [10, 4, 16, 12, 8]</span>
<span class="err">Variável: retorno | Tipo: <class 'list'> | Valor: [10, 4, 16, 12, 8]</span>
</pre></div>
<p>Nesse caso, pode-se apenas chamar a função, como também utilizá-la nos argumentos de outras funções. Porém para se ter os valores originais, deve-se fazer uma cópia manualmente antes de executar a função.</p>
<h2>Exemplo na biblioteca padrão</h2>
<p>Na biblioteca padrão do Python, existem os métodos <code>sort</code> e <code>reverse</code> que seguem o padrão <em>in place</em>, e as funções <code>sorted</code> e <code>reversed</code> que seguem o padrão cópia de valor, podendo ser utilizados para ordenar e inverter os valores de uma lista, por exemplo. Quando não é mais necessário uma cópia da lista com a ordem original, é preferível utilizar funções <em>in place</em>, que alteram a própria lista, e como não criam uma cópia da lista, utilizam menos memória. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">valores</span><span class="o">.</span><span class="n">sort</span><span class="p">()</span>
<span class="n">valores</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
</pre></div>
<p>Se for necessário manter uma cópia da lista inalterada, deve-se optar pelas funções de cópia de valor. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">novos_valores</span> <span class="o">=</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">valores</span><span class="p">))</span>
<span class="nb">print</span><span class="p">(</span><span class="n">novos_valores</span><span class="p">)</span>
</pre></div>
<p>Porém esse exemplo cria duas cópias da lista, uma em cada função. Para criar apenas uma cópia, pode-se misturar funções <em>in place</em> com cópia de valor. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">novos_valores</span> <span class="o">=</span> <span class="nb">sorted</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
<span class="n">novos_valores</span><span class="o">.</span><span class="n">reverse</span><span class="p">()</span>
<span class="nb">print</span><span class="p">(</span><span class="n">novos_valores</span><span class="p">)</span>
</pre></div>
<p>Também vale observar que algumas utilizações dessas funções podem dar a impressão de que elas não funcionaram, como:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="nb">sorted</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span>
<span class="nb">print</span><span class="p">(</span><span class="n">valores</span><span class="p">)</span> <span class="c1"># Imprime a lista original, e não a ordenada</span>
<span class="nb">print</span><span class="p">(</span><span class="n">valores</span><span class="o">.</span><span class="n">sort</span><span class="p">())</span> <span class="c1"># Imprime None e não a lista</span>
</pre></div>
<h2>Considerações</h2>
<p>Nem sempre é possível utilizar o padrão desejado, <em>strings</em> no Python (<code>str</code>) são imutáveis, logo todas as funções que manipulam elas seguiram o padrão cópia de valor, e para outros tipos, pode ocorrer de só existir funções <em>in place</em>, sendo necessário fazer uma cópia manualmente antes de chamar a função, caso necessário. Para saber qual padrão a função implementa, é necessário consultar sua documentação, ou verificando sua assinatura, embora ainda possa existir uma dúvida entre cópia de valor e híbrida, visto que a assinatura dos dois padrões são iguais.</p>
<p>Os exemplos aqui dados são didáticos. Caso deseja-se ordenar de forma reversa, tanto o método <code>sort</code>, quanto a função <code>sorted</code> podem receber como argumento <code>reverse=True</code>, e assim já fazer a ordenação reversa. Assim como é possível criar uma nova lista já com os valores, sem precisar adicionar manualmente item por item, como os exemplos:</p>
<div class="highlight"><pre><span></span><span class="n">valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">5</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">8</span><span class="p">,</span> <span class="mi">6</span><span class="p">,</span> <span class="mi">4</span><span class="p">]</span>
<span class="n">partes_dos_valores</span> <span class="o">=</span> <span class="n">valores</span><span class="p">[</span><span class="mi">2</span><span class="p">:]</span>
<span class="n">novos_valores</span> <span class="o">=</span> <span class="p">[</span><span class="mi">2</span> <span class="o">*</span> <span class="n">valor</span> <span class="k">for</span> <span class="n">valor</span> <span class="ow">in</span> <span class="n">valores</span><span class="p">]</span>
</pre></div>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Encapsulamento da lógica do algoritmotag:pythonclub.com.br,2021-03-02:/encapsulamento-da-logica-do-algoritmo.html2021-03-02T18:00:00+00:00<p>Muitas listas de exercícios de lógica de programação pedem em algum momento que um valor seja lido do teclado, e caso esse valor seja inválido, deve-se avisar, e repetir a leitura até que um valor válido seja informado. Utilizando a ideia de <a href="https://dev.to/acaverna/otimizando-o-algoritmo-passo-a-passo-4co0">otimização do algoritmo passo a passo</a>, começando com uma solução simples, pretendo estudar como reduzir a duplicação de código alterando o algoritmo, encapsulando a lógica em funções, e encapsulando em classes.</p>
<h2>Exercício</h2>
<p>Um exemplo de exercício que pede esse tipo de validação é a leitura de notas, que devem estar entre 0 e 10. A solução mais simples, consiste em ler um valor, e enquanto esse valor for inválido, dar o aviso e ler outro valor. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="n">nota</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">))</span>
<span class="k">while</span> <span class="n">nota</span> <span class="o"><</span> <span class="mi">0</span> <span class="ow">or</span> <span class="n">nota</span> <span class="o">></span> <span class="mi">10</span><span class="p">:</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Nota inválida'</span><span class="p">)</span>
<span class="n">nota</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">))</span>
</pre></div>
<p>Esse algoritmo funciona, porém existe uma duplicação no código que faz a leitura da nota (uma antes do <em>loop</em> e outra dentro). Caso seja necessário uma alteração, como a mudança da nota para um valor inteiro entre 0 e 100, deve-se alterar os dois lugares, e se feito em apenas um lugar, o algoritmo poderia processar valores inválidos.</p>
<h3>Alterando o algoritmo</h3>
<p>Visando remover a repetição de código, é possível unificar a leitura do valor dentro do <em>loop</em>, uma vez que é necessário repetir essa instrução até que o valor válido seja obtido. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="n">nota</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">))</span>
<span class="k">if</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">nota</span> <span class="o"><=</span> <span class="mi">10</span><span class="p">:</span>
<span class="k">break</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Nota inválida!'</span><span class="p">)</span>
</pre></div>
<p>Dessa forma, não existe mais a repetição de código. A condição de parada, que antes verificava se o valor era inválido (o que pode ter uma leitura não tão intuitiva), agora verifica se é um valor válido (que é geralmente é mais fácil de ler e escrever a condição). E a ordem dos comandos dentro do <em>loop</em>, que agora estão em uma ordem que facilita a leitura, visto que no algoritmo anterior era necessário tem em mente o que era executado antes do <em>loop</em>.</p>
<p>Porém esses algoritmos validam apenas o valor lido, apresentando erro caso seja informado um valor com formato inválido, como letras em vez de números. Isso pode ser resolvido tratando as exceções lançadas. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">nota</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">))</span>
<span class="k">if</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">nota</span> <span class="o"><=</span> <span class="mi">10</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Nota inválida!'</span><span class="p">)</span>
</pre></div>
<h3>Encapsulamento da lógica em função</h3>
<p>Caso fosse necessário ler várias notas, com os algoritmos apresentados até então, seria necessário repetir todo esse trecho de código, ou utilizá-lo dentro de uma estrutura de repetição. Para facilitar sua reutilização, evitando a duplicação de código, é possível encapsular esse algoritmo dentro de uma função. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">nota_input</span><span class="p">(</span><span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">nota</span> <span class="o">=</span> <span class="nb">float</span><span class="p">(</span><span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">nota</span> <span class="o"><=</span> <span class="mi">10</span><span class="p">:</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="s1">'Nota inválida!'</span><span class="p">)</span>
<span class="k">return</span> <span class="n">nota</span>
<span class="n">nota1</span> <span class="o">=</span> <span class="n">nota_input</span><span class="p">(</span><span class="s1">'Digite a primeira nota: '</span><span class="p">)</span>
<span class="n">nota2</span> <span class="o">=</span> <span class="n">nota_input</span><span class="p">(</span><span class="s1">'Digite a segunda nota: '</span><span class="p">)</span>
</pre></div>
<h3>Encapsulamento da lógica em classes</h3>
<p>Em vez de encapsular essa lógica em uma função, é possível encapsulá-la em uma classe, o que permitiria separar cada etapa do algoritmo em métodos, assim como ter um método responsável por controlar qual etapa deveria ser chamada em qual momento. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ValidaNotaInput</span><span class="p">:</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nota inválida!'</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">entrada</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">validar_nota</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">nota</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">nota</span> <span class="o"><=</span> <span class="mi">10</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">nota</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_nota</span><span class="p">(</span><span class="n">nota</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">nota</span>
<span class="n">nota_input</span> <span class="o">=</span> <span class="n">ValidaNotaInput</span><span class="p">()</span>
<span class="n">nota</span> <span class="o">=</span> <span class="n">nota_input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">)</span>
</pre></div>
<p>Vale observar que o método <code>__call__</code> permite que o objeto criado a partir dessa classe seja chamado como se fosse uma função. Nesse caso ele é o responsável por chamar cada etapa do algoritmo, como: <code>ler_entrada</code> que é responsável por ler o que foi digitado no teclado, <code>transformar_entrada</code> que é responsável por converter o texto lido para o tipo desejado (converter de <code>str</code> para <code>float</code>), e <code>validar_nota</code> que é responsável por dizer se o valor é válido ou não. Vale observar que ao dividir o algoritmo em métodos diferentes, seu código principal virou uma espécie de código comentado, descrevendo o que está sendo feito e onde está sendo feito.</p>
<p>Outra vantagem de encapsular a lógica em classe, em vez de uma função, é a possibilidade de generalizá-la. Se fosse necessário validar outro tipo de entrada, encapsulando em uma função, seria necessário criar outra função repetindo todo o algoritmo, alterando apenas a parte referente a transformação do valor lido, e validação, o que gera uma espécie de repetição de código. Ao encapsular em classes, é possível se aproveitar dos mecanismos de herança para evitar essa repetição. Exemplo:</p>
<div class="highlight"><pre><span></span><span class="k">class</span> <span class="nc">ValidaInput</span><span class="p">:</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Valor inválido!'</span>
<span class="k">def</span> <span class="nf">ler_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">input</span><span class="p">(</span><span class="n">prompt</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">raise</span> <span class="ne">NotImplementedError</span>
<span class="k">def</span> <span class="fm">__call__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">prompt</span><span class="p">):</span>
<span class="k">while</span> <span class="kc">True</span><span class="p">:</span>
<span class="k">try</span><span class="p">:</span>
<span class="n">valor</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">ler_entrada</span><span class="p">(</span><span class="n">prompt</span><span class="p">))</span>
<span class="k">if</span> <span class="bp">self</span><span class="o">.</span><span class="n">validar_valor</span><span class="p">(</span><span class="n">valor</span><span class="p">):</span>
<span class="k">break</span>
<span class="k">except</span> <span class="ne">ValueError</span><span class="p">:</span>
<span class="o">...</span>
<span class="nb">print</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">mensagem_valor_invalido</span><span class="p">)</span>
<span class="k">return</span> <span class="n">valor</span>
<span class="k">class</span> <span class="nc">ValidaNomeInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nome inválido!'</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="n">entrada</span><span class="o">.</span><span class="n">strip</span><span class="p">()</span><span class="o">.</span><span class="n">title</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="n">valor</span> <span class="o">!=</span> <span class="s1">''</span>
<span class="k">class</span> <span class="nc">ValidaNotaInput</span><span class="p">(</span><span class="n">ValidaInput</span><span class="p">):</span>
<span class="n">mensagem_valor_invalido</span> <span class="o">=</span> <span class="s1">'Nota inválida!'</span>
<span class="k">def</span> <span class="nf">transformar_entrada</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">entrada</span><span class="p">):</span>
<span class="k">return</span> <span class="nb">float</span><span class="p">(</span><span class="n">entrada</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">validar_valor</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">valor</span><span class="p">):</span>
<span class="k">return</span> <span class="mi">0</span> <span class="o"><=</span> <span class="n">valor</span> <span class="o"><=</span> <span class="mi">10</span>
<span class="n">nome_input</span> <span class="o">=</span> <span class="n">ValidaNomeInput</span><span class="p">()</span>
<span class="n">nota_input</span> <span class="o">=</span> <span class="n">ValidaNotaInput</span><span class="p">()</span>
<span class="n">nome</span> <span class="o">=</span> <span class="n">nome_input</span><span class="p">(</span><span class="s1">'Digite o nome: '</span><span class="p">)</span>
<span class="n">nota</span> <span class="o">=</span> <span class="n">nota_input</span><span class="p">(</span><span class="s1">'Digite a nota: '</span><span class="p">)</span>
</pre></div>
<p>Dessa forma, é possível reutilizar o código já existente para criar outras validações, sendo necessário implementar apenas como converter a <code>str</code> lida do teclado para o tipo desejado, e como esse valor deve ser validado. Não é necessário entender e repetir a lógica de ler o valor, validá-lo, imprimir a mensagem de erro, e repetir até que seja informado um valor válido.</p>
<h2>Considerações</h2>
<p>É possível encapsular a lógica de um algoritmo em funções ou em classes. Embora para fazê-lo em uma classe exija conhecimentos de programação orientada a objetos, o seu reaproveitamento é facilitado, abstraindo toda a complexidade do algoritmo, que pode ser disponibilizado através de uma biblioteca, exigindo apenas a implementações de métodos simples por quem for a utilizar.</p>
<p>Ainda poderia ser discutido outras formas de fazer essa implementação, como passar funções como parâmetro e a utilização de <a href="https://docs.python.org/pt-br/3/library/asyncio-task.html">corrotinas</a> no encapsulamento do algoritmo em função, assim como a utilização de <a href="https://docs.python.org/pt-br/3/library/functions.html#classmethod">classmethod</a>, <a href="https://docs.python.org/pt-br/3/library/functions.html#staticmethod">staticmethod</a> e <a href="https://docs.python.org/pt-br/3/library/abc.html">ABC</a> no encapsulamento do algoritmo em classes.</p>
<hr />
<p>Esse artigo foi publicado originalmente no <a href="https://eduardoklosowski.github.io/blog/">meu blog</a>, passe por lá, ou siga-me no <a href="https://dev.to/eduardoklosowski">DEV</a> para ver mais artigos que eu escrevi.</p>Eduardo Klosowskihttp://pythonclub.com.br/Blog has moved!tag:blogger.com,1999:blog-4820627057316560708.post-25547210874096695282021-02-07T23:41:22+00:00It's 2021 and I decided to start blogging again. I am not going to migrate the posts to the new blog nor disable this blog for now, but if you want to checkout my latest content, please go to <a href="https://blog.fsouza.dev">https://blog.fsouza.dev</a>! :)fsouzanoreply@blogger.comhttp://f.souza.cc/Mocando um serviço com Bottlehttps://paulohrpinheiro.xyz/texts/python/2021-01-11-mocando-um-servico-com-bottle.html2021-01-11T00:00:00+00:00Como minha vida ficou mais fácil com o Bottle ou, uma linda história de NatalBlog do PauloHRPinheirohttps://paulohrpinheiro.xyzHosting Telegram bots on Cloud Run for freehttps://nullonerror.org/2021/01/08/hosting-telegram-bots-on-google-cloud-run2021-01-08T00:00:00+00:00<p>I write a lot of <a href="https://core.telegram.org/bots">Telegram bots</a> using the library <a href="https://github.com/python-telegram-bot/python-telegram-bot">python-telegram-bot</a>. Writing Telegram bots is fun, but you will also need someplace to host them.</p>
<p>I personally like the new <a href="https://cloud.google.com/run">Google Cloud Run</a>; or run, for short, is perfect because it has a <em>“gorgeous”</em> <a href="https://cloud.google.com/run/pricing">free quota</a> that should be mostly sufficient to host your bots, also, and is it super simple to deploy and get running.</p>
<p>To create Telegram bots, first, you need to talk to <a href="https://t.me/botfather">BotFather</a> and get a <em>TOKEN</em>.</p>
<p>Secondly, you need some coding. As I mentioned before, you can use <em>python-telegram-bot</em> to do your bots. Here is the <a href="https://python-telegram-bot.org/">documentation</a>.</p>
<h3 id="code">Code</h3>
<p>Here is the base code that you will need to run on Cloud Run.</p>
<p><code class="language-plaintext highlighter-rouge">main.py</code></p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">http</span>
<span class="kn">from</span> <span class="nn">flask</span> <span class="kn">import</span> <span class="n">Flask</span><span class="p">,</span> <span class="n">request</span>
<span class="kn">from</span> <span class="nn">werkzeug.wrappers</span> <span class="kn">import</span> <span class="n">Response</span>
<span class="kn">from</span> <span class="nn">telegram</span> <span class="kn">import</span> <span class="n">Bot</span><span class="p">,</span> <span class="n">Update</span>
<span class="kn">from</span> <span class="nn">telegram.ext</span> <span class="kn">import</span> <span class="n">Dispatcher</span><span class="p">,</span> <span class="n">Filters</span><span class="p">,</span> <span class="n">MessageHandler</span><span class="p">,</span> <span class="n">CallbackContext</span>
<span class="n">app</span> <span class="o">=</span> <span class="n">Flask</span><span class="p">(</span><span class="n">__name__</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">echo</span><span class="p">(</span><span class="n">update</span><span class="p">:</span> <span class="n">Update</span><span class="p">,</span> <span class="n">context</span><span class="p">:</span> <span class="n">CallbackContext</span><span class="p">)</span> <span class="o">-></span> <span class="bp">None</span><span class="p">:</span>
<span class="n">update</span><span class="p">.</span><span class="n">message</span><span class="p">.</span><span class="n">reply_text</span><span class="p">(</span><span class="n">update</span><span class="p">.</span><span class="n">message</span><span class="p">.</span><span class="n">text</span><span class="p">)</span>
<span class="n">bot</span> <span class="o">=</span> <span class="n">Bot</span><span class="p">(</span><span class="n">token</span><span class="o">=</span><span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">[</span><span class="s">"TOKEN"</span><span class="p">])</span>
<span class="n">dispatcher</span> <span class="o">=</span> <span class="n">Dispatcher</span><span class="p">(</span><span class="n">bot</span><span class="o">=</span><span class="n">bot</span><span class="p">,</span> <span class="n">update_queue</span><span class="o">=</span><span class="bp">None</span><span class="p">,</span> <span class="n">workers</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="n">dispatcher</span><span class="p">.</span><span class="n">add_handler</span><span class="p">(</span><span class="n">MessageHandler</span><span class="p">(</span><span class="n">Filters</span><span class="p">.</span><span class="n">text</span> <span class="o">&</span> <span class="o">~</span><span class="n">Filters</span><span class="p">.</span><span class="n">command</span><span class="p">,</span> <span class="n">echo</span><span class="p">))</span>
<span class="o">@</span><span class="n">app</span><span class="p">.</span><span class="n">route</span><span class="p">(</span><span class="s">"/"</span><span class="p">,</span> <span class="n">methods</span><span class="o">=</span><span class="p">[</span><span class="s">"POST"</span><span class="p">])</span>
<span class="k">def</span> <span class="nf">index</span><span class="p">()</span> <span class="o">-></span> <span class="n">Response</span><span class="p">:</span>
<span class="n">dispatcher</span><span class="p">.</span><span class="n">process_update</span><span class="p">(</span>
<span class="n">Update</span><span class="p">.</span><span class="n">de_json</span><span class="p">(</span><span class="n">request</span><span class="p">.</span><span class="n">get_json</span><span class="p">(</span><span class="n">force</span><span class="o">=</span><span class="bp">True</span><span class="p">),</span> <span class="n">bot</span><span class="p">))</span>
<span class="k">return</span> <span class="s">""</span><span class="p">,</span> <span class="n">http</span><span class="p">.</span><span class="n">HTTPStatus</span><span class="p">.</span><span class="n">NO_CONTENT</span>
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">requirements.txt</code></p>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>flask==1.1.2
gunicorn==20.0.4
python-telegram-bot==13.1
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">Dockerfile</code></p>
<div class="language-dockerfile highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span><span class="s"> python:3.8-slim</span>
<span class="k">ENV</span><span class="s"> PYTHONUNBUFFERED True</span>
<span class="k">WORKDIR</span><span class="s"> /app</span>
<span class="k">COPY</span><span class="s"> *.txt .</span>
<span class="k">RUN </span>pip <span class="nb">install</span> <span class="nt">--no-cache-dir</span> <span class="nt">--upgrade</span> pip <span class="nt">-r</span> requirements.txt
<span class="k">COPY</span><span class="s"> . ./</span>
<span class="k">CMD</span><span class="s"> exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app</span>
</code></pre></div></div>
<h3 id="deployment">Deployment</h3>
<p>Finally, you need to deploy. You can do it in a single step, but first, let’s run the command below to set the default region (optionally).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud config <span class="nb">set </span>run/region us-central1
</code></pre></div></div>
<p>Then deploy to Cloud Run:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud beta run deploy your-bot-name <span class="se">\</span>
<span class="nt">--source</span> <span class="nb">.</span> <span class="se">\</span>
<span class="nt">--set-env-vars</span> <span class="nv">TOKEN</span><span class="o">=</span>your-telegram-bot-token <span class="se">\</span>
<span class="nt">--platform</span> managed <span class="se">\</span>
<span class="nt">--allow-unauthenticated</span> <span class="se">\</span>
<span class="nt">--project</span> your-project-name
</code></pre></div></div>
<p>After this, you will receive a public <em>URL</em> of your run, and you will need to set the Telegram bot <code class="language-plaintext highlighter-rouge">webHook</code> using <em>cURL</em></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="s2">"https://api.telegram.org/botYOUR-BOT:TOKEN/setWebhook?url=https://your-bot-name-uuid-uc.a.run.app"</span>
</code></pre></div></div>
<p>You should replace the <code class="language-plaintext highlighter-rouge">YOUR-BOT:TOKEN</code> by the bot’s token and the public URL of your Cloud Run.</p>
<p>This should be enough.</p><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/NullOnError?a=GH9gZi9oKv0:Nyu2eTygd0k:yIl2AUoC8zA"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=yIl2AUoC8zA" /></a> <a href="http://feeds.feedburner.com/~ff/NullOnError?a=GH9gZi9oKv0:Nyu2eTygd0k:qj6IDK7rITs"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=qj6IDK7rITs" /></a>
</div><img alt="" height="1" src="http://feeds.feedburner.com/~r/NullOnError/~4/GH9gZi9oKv0" width="1" />I write a lot of Telegram bots using the library python-telegram-bot. Writing Telegram bots is fun, but you will also need someplace to host them.Rodrigo Delducarodrigodelduca@gmail.comhttps://nullonerror.org/Periodically backup your Google Photos to Google Cloud Storagehttps://nullonerror.org/2020/12/31/backup-your-google-photos-on-cloud2020-12-31T00:00:00+00:00<h3 id="why">Why?</h3>
<p>Google Cloud Storage is cheaper, and you pay only for what you use than <a href="https://one.google.com/">Google One</a>. Also, you can erase any photo, and you still have a copy of that.</p>
<h3 id="installation">Installation</h3>
<p>Create a Compute Engine (a VM).</p>
<p>If you choose Ubuntu, first of all, remove <code class="language-plaintext highlighter-rouge">snap</code></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt autoremove <span class="nt">--purge</span> snapd
<span class="nb">sudo rm</span> <span class="nt">-rf</span> /var/cache/snapd/
<span class="nb">rm</span> <span class="nt">-rf</span> ~/snap
</code></pre></div></div>
<p>Install <code class="language-plaintext highlighter-rouge">gcsfuse</code> or follow <a href="https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md">the official instructions</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">GCSFUSE_REPO</span><span class="o">=</span>gcsfuse-<span class="sb">`</span>lsb_release <span class="nt">-c</span> <span class="nt">-s</span><span class="sb">`</span>
<span class="nb">echo</span> <span class="s2">"deb http://packages.cloud.google.com/apt </span><span class="nv">$GCSFUSE_REPO</span><span class="s2"> main"</span> | <span class="nb">sudo tee</span> /etc/apt/sources.list.d/gcsfuse.list
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | <span class="nb">sudo </span>apt-key add -
<span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get <span class="nb">install </span>gcsfuse
</code></pre></div></div>
<p>On Google Cloud console create a bucket of the type <code class="language-plaintext highlighter-rouge">Nearline</code>, in my case the name of the bucket is <code class="language-plaintext highlighter-rouge">tank1</code>, then back to your VM and create a dir with the same name of the bucket.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>name-of-your-bucket
</code></pre></div></div>
<p>Now install <code class="language-plaintext highlighter-rouge">gphotos-sync</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt <span class="nb">install</span> <span class="nt">-y</span> python3-pip
pip3 <span class="nb">install </span>gphotos-sync
</code></pre></div></div>
<p>I created a small Python script to deal with multiple Google accounts. I’ll explain later how it works.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">cat</span> <span class="o"><<</span><span class="n">EOF</span> <span class="o">></span> <span class="o">/</span><span class="n">home</span><span class="o">/</span><span class="n">ubuntu</span><span class="o">/</span><span class="n">synchronize</span><span class="p">.</span><span class="n">py</span>
<span class="c1">#!/usr/bin/env python3
</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">from</span> <span class="nn">pathlib</span> <span class="kn">import</span> <span class="n">Path</span>
<span class="kn">import</span> <span class="nn">requests</span>
<span class="n">home</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="n">os</span><span class="p">.</span><span class="n">path</span><span class="p">.</span><span class="n">expanduser</span><span class="p">(</span><span class="s">"~"</span><span class="p">))</span> <span class="o">/</span> <span class="s">"tank1/photos"</span>
<span class="n">args</span> <span class="o">=</span> <span class="p">[</span>
<span class="s">"--ntfs"</span><span class="p">,</span>
<span class="s">"--retry-download"</span><span class="p">,</span>
<span class="s">"--skip-albums"</span><span class="p">,</span>
<span class="s">"--photos-path"</span><span class="p">,</span> <span class="s">"."</span><span class="p">,</span>
<span class="s">"--log-level"</span><span class="p">,</span> <span class="s">"DEBUG"</span><span class="p">,</span>
<span class="p">]</span>
<span class="n">env</span> <span class="o">=</span> <span class="n">os</span><span class="p">.</span><span class="n">environ</span><span class="p">.</span><span class="n">copy</span><span class="p">()</span>
<span class="n">env</span><span class="p">[</span><span class="s">"LC_ALL"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"en_US.UTF-8"</span>
<span class="k">for</span> <span class="n">p</span> <span class="ow">in</span> <span class="n">home</span><span class="p">.</span><span class="n">glob</span><span class="p">(</span><span class="s">"*/*"</span><span class="p">):</span>
<span class="n">subprocess</span><span class="p">.</span><span class="n">run</span><span class="p">([</span><span class="s">"/home/ubuntu/.local/bin/gphotos-sync"</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="nb">str</span><span class="p">(</span><span class="n">p</span><span class="p">.</span><span class="n">relative_to</span><span class="p">(</span><span class="n">home</span><span class="p">))],</span> <span class="n">check</span><span class="o">=</span><span class="bp">True</span><span class="p">,</span> <span class="n">cwd</span><span class="o">=</span><span class="n">home</span><span class="p">,</span> <span class="n">env</span><span class="o">=</span><span class="n">env</span><span class="p">,</span> <span class="n">stdout</span><span class="o">=</span><span class="n">sys</span><span class="p">.</span><span class="n">stdout</span><span class="p">,</span> <span class="n">stderr</span><span class="o">=</span><span class="n">subprocess</span><span class="p">.</span><span class="n">STDOUT</span><span class="p">)</span>
<span class="c1"># I use healthchecks.io to alert me if the script has stopped work
</span><span class="n">url</span> <span class="o">=</span> <span class="s">"https://hc-ping.com/uuid4"</span>
<span class="n">response</span> <span class="o">=</span> <span class="n">requests</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="mi">60</span><span class="p">)</span>
<span class="n">response</span><span class="p">.</span><span class="n">raise_for_status</span><span class="p">()</span>
<span class="n">EOF</span>
</code></pre></div></div>
<p>Give <em>execute</em> permission.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>chmod u+x synchronize.py
</code></pre></div></div>
<p>Now let’s create some <em>systemd</em> scripts.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo su
</code></pre></div></div>
<p>Let’s create a service to gcsfuse, responsible to mount the bucket locally using the FUSE.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o"><<</span><span class="no">EOF</span><span class="sh"> >/etc/systemd/system/gcsfuse.service
# Script stolen from https://gist.github.com/craigafinch/292f98618f8eadc33e9633e6e3b54c05
[Unit]
Description=Google Cloud Storage FUSE mounter
After=local-fs.target network-online.target google.service sys-fs-fuse-connections.mount
Before=shutdown.target
[Service]
Type=forking
User=ubuntu
ExecStart=/bin/gcsfuse tank1 /home/ubuntu/tank1
ExecStop=/bin/fusermount -u /home/ubuntu/tank1
Restart=always
[Install]
WantedBy=multi-user.target
</span><span class="no">EOF
</span></code></pre></div></div>
<p>Enable and start the service:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl <span class="nb">enable </span>gcsfuse.service
systemctl start gcsfuse.service
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o"><<</span><span class="no">EOF</span><span class="sh"> >/etc/systemd/system/gphotos-sync.service
[Unit]
Description=Run gphotos-sync for each account
[Service]
User=ubuntu
ExecStart=/home/ubuntu/synchronize.py
</span><span class="no">EOF
</span></code></pre></div></div>
<p>And enable the service.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl <span class="nb">enable </span>gphotos-sync.service
</code></pre></div></div>
<p>Now let’s create a <em>timer</em> to run 1 minute after the boot the <code class="language-plaintext highlighter-rouge">gphotos-sync.service</code> with <code class="language-plaintext highlighter-rouge">gcsfuse.service</code> as dependency.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> <span class="o"><<</span><span class="no">EOF</span><span class="sh"> >/etc/systemd/system/gphotos-sync.timer
[Unit]
Description=Run gphotos sync service weekly
Requires=gcsfuse.service
[Timer]
OnBootSec=1min
Unit=gphotos-sync.service
[Install]
WantedBy=timers.target
</span><span class="no">EOF
</span></code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl <span class="nb">enable </span>gphotos-sync.timer
systemctl start gphotos-sync.timer
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">exit</code> (back to ubuntu user)</p>
<p>Now follow <a href="https://nullonerror.org/2020/12/31/backup-your-google-photos-on-cloud/this instructions">https://docs.google.com/document/d/1ck1679H8ifmZ_4eVbDeD_-jezIcZ-j6MlaNaeQiz7y0/edit</a> to get a <code class="language-plaintext highlighter-rouge">client_secret.json</code> to use with <code class="language-plaintext highlighter-rouge">gphotos-sync</code>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> /home/ubuntu/.config/gphotos-sync/
<span class="c"># Copy the contents of the json to the file bellow</span>
vim /home/ubuntu/.config/gphotos-sync/client_secret.json
</code></pre></div></div>
<h3 id="testing">Testing</h3>
<p>Due to an issue with <code class="language-plaintext highlighter-rouge">gcsfuse</code>, I was unable to create the backup dir directly on the bucket. The workaround is to create a <em>temp</em> directory and start the <code class="language-plaintext highlighter-rouge">gphotos-sync</code> manually first.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> ~/temp/username/0
<span class="nb">cd</span> ~/temp
gphotos-sync <span class="nt">--ntfs</span> <span class="nt">--skip-albums</span> <span class="nt">--photos-path</span> <span class="nb">.</span> username/0
<span class="c"># gphotos-sync will ask for a token, paste it and CTRL-C to stop the download of photos.</span>
<span class="nb">cp</span> ~/temp/username/ ~/tank1/photos/username
</code></pre></div></div>
<p>Verify if it is working.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>./synchronize.py
</code></pre></div></div>
<p>After executing the command above, the script should start the backup. You can wait until it finishes or continue to the steps below.</p>
<h3 id="schedule-startup-and-shutdown-of-the-vm">Schedule startup and shutdown of the VM</h3>
<p>The content below is based on and simplified version of <a href="https://cloud.google.com/scheduler/docs/start-and-stop-compute-engine-instances-on-a-schedule#gcloud_3">Scheduling compute instances with Cloud Scheduler by Google</a></p>
<p>Back to your VM and add the label <code class="language-plaintext highlighter-rouge">runtime</code> with the value <code class="language-plaintext highlighter-rouge">weekly</code>, this is needed by the <em>function</em> below to know which instances should be started or shutdown.</p>
<p>Create a new directory, in my case, I will call <code class="language-plaintext highlighter-rouge">functions</code> and add two files:</p>
<p><code class="language-plaintext highlighter-rouge">index.js</code></p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">Compute</span> <span class="o">=</span> <span class="nx">require</span><span class="p">(</span><span class="dl">'</span><span class="s1">@google-cloud/compute</span><span class="dl">'</span><span class="p">);</span>
<span class="kd">const</span> <span class="nx">compute</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Compute</span><span class="p">();</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">startInstancePubSub</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">payload</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">).</span><span class="nx">toString</span><span class="p">());</span>
<span class="kd">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="p">{</span><span class="na">filter</span><span class="p">:</span> <span class="s2">`labels.</span><span class="p">${</span><span class="nx">payload</span><span class="p">.</span><span class="nx">label</span><span class="p">}</span><span class="s2">`</span><span class="p">};</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">vms</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">compute</span><span class="p">.</span><span class="nx">getVMs</span><span class="p">(</span><span class="nx">options</span><span class="p">);</span>
<span class="k">await</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span>
<span class="nx">vms</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="k">async</span> <span class="nx">instance</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">payload</span><span class="p">.</span><span class="nx">zone</span> <span class="o">===</span> <span class="nx">instance</span><span class="p">.</span><span class="nx">zone</span><span class="p">.</span><span class="nx">id</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">operation</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">compute</span>
<span class="p">.</span><span class="nx">zone</span><span class="p">(</span><span class="nx">payload</span><span class="p">.</span><span class="nx">zone</span><span class="p">)</span>
<span class="p">.</span><span class="nx">vm</span><span class="p">(</span><span class="nx">instance</span><span class="p">.</span><span class="nx">name</span><span class="p">)</span>
<span class="p">.</span><span class="nx">start</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">operation</span><span class="p">.</span><span class="nx">promise</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">);</span>
<span class="kd">const</span> <span class="nx">message</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">Successfully started instance(s)</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">message</span><span class="p">);</span>
<span class="nx">callback</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="nx">message</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
<span class="nx">callback</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
<span class="nx">exports</span><span class="p">.</span><span class="nx">stopInstancePubSub</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">event</span><span class="p">,</span> <span class="nx">context</span><span class="p">,</span> <span class="nx">callback</span><span class="p">)</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">try</span> <span class="p">{</span>
<span class="kd">const</span> <span class="nx">payload</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="nx">event</span><span class="p">.</span><span class="nx">data</span><span class="p">,</span> <span class="dl">'</span><span class="s1">base64</span><span class="dl">'</span><span class="p">).</span><span class="nx">toString</span><span class="p">());</span>
<span class="kd">const</span> <span class="nx">options</span> <span class="o">=</span> <span class="p">{</span><span class="na">filter</span><span class="p">:</span> <span class="s2">`labels.</span><span class="p">${</span><span class="nx">payload</span><span class="p">.</span><span class="nx">label</span><span class="p">}</span><span class="s2">`</span><span class="p">};</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">vms</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">compute</span><span class="p">.</span><span class="nx">getVMs</span><span class="p">(</span><span class="nx">options</span><span class="p">);</span>
<span class="k">await</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">all</span><span class="p">(</span>
<span class="nx">vms</span><span class="p">.</span><span class="nx">map</span><span class="p">(</span><span class="k">async</span> <span class="nx">instance</span> <span class="o">=></span> <span class="p">{</span>
<span class="k">if</span> <span class="p">(</span><span class="nx">payload</span><span class="p">.</span><span class="nx">zone</span> <span class="o">===</span> <span class="nx">instance</span><span class="p">.</span><span class="nx">zone</span><span class="p">.</span><span class="nx">id</span><span class="p">)</span> <span class="p">{</span>
<span class="kd">const</span> <span class="p">[</span><span class="nx">operation</span><span class="p">]</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">compute</span>
<span class="p">.</span><span class="nx">zone</span><span class="p">(</span><span class="nx">payload</span><span class="p">.</span><span class="nx">zone</span><span class="p">)</span>
<span class="p">.</span><span class="nx">vm</span><span class="p">(</span><span class="nx">instance</span><span class="p">.</span><span class="nx">name</span><span class="p">)</span>
<span class="p">.</span><span class="nx">stop</span><span class="p">();</span>
<span class="k">return</span> <span class="nx">operation</span><span class="p">.</span><span class="nx">promise</span><span class="p">();</span>
<span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
<span class="k">return</span> <span class="nb">Promise</span><span class="p">.</span><span class="nx">resolve</span><span class="p">();</span>
<span class="p">}</span>
<span class="p">})</span>
<span class="p">);</span>
<span class="kd">const</span> <span class="nx">message</span> <span class="o">=</span> <span class="dl">'</span><span class="s1">Successfully stopped instance(s)</span><span class="dl">'</span><span class="p">;</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">message</span><span class="p">);</span>
<span class="nx">callback</span><span class="p">(</span><span class="kc">null</span><span class="p">,</span> <span class="nx">message</span><span class="p">);</span>
<span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">err</span><span class="p">)</span> <span class="p">{</span>
<span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
<span class="nx">callback</span><span class="p">(</span><span class="nx">err</span><span class="p">);</span>
<span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>
<p>And</p>
<p><code class="language-plaintext highlighter-rouge">package.json</code></p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nl">"main"</span><span class="p">:</span><span class="w"> </span><span class="s2">"index.js"</span><span class="p">,</span><span class="w">
</span><span class="nl">"private"</span><span class="p">:</span><span class="w"> </span><span class="kc">true</span><span class="p">,</span><span class="w">
</span><span class="nl">"dependencies"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nl">"@google-cloud/compute"</span><span class="p">:</span><span class="w"> </span><span class="s2">"^2.4.1"</span><span class="w">
</span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>
<p>Create a <em>PubSub</em> topic to start the instance.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud pubsub topics create start-instance-event
</code></pre></div></div>
<p>Now deploy the <code class="language-plaintext highlighter-rouge">startInstancePubSub</code> function</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud functions deploy startInstancePubSub <span class="se">\</span>
<span class="nt">--trigger-topic</span> start-instance-event <span class="se">\</span>
<span class="nt">--runtime</span> nodejs12 <span class="se">\</span>
<span class="nt">--allow-unauthenticated</span>
</code></pre></div></div>
<p>And another <em>PubSub</em> topic to stop the instance.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud pubsub topics create stop-instance-event
</code></pre></div></div>
<p>And the <code class="language-plaintext highlighter-rouge">stopInstancePubSub</code> function</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud functions deploy stopInstancePubSub <span class="se">\</span>
<span class="nt">--trigger-topic</span> stop-instance-event <span class="se">\</span>
<span class="nt">--runtime</span> nodejs12 <span class="se">\</span>
<span class="nt">--allow-unauthenticated</span>
</code></pre></div></div>
<p>And finally, let’s create two <em>Cloud Scheduler</em> to publish on the topics on <em>Sunday</em> and <em>Monday</em> at midnight.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud beta scheduler jobs create pubsub startup-weekly-instances \
--schedule '0 0 * * SUN' \
--topic start-instance-event \
--message-body '{"zone":"us-central1-a", "label":"runtime=weekly"}' \
--time-zone 'America/Sao_Paulo'
</code></pre></div></div>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud beta scheduler jobs create pubsub shutdown-weekly-instances \
--schedule '0 0 * * MON' \
--topic stop-instance-event \
--message-body '{"zone":"us-central1-a", "label":"runtime=weekly"}' \
--time-zone 'America/Sao_Paulo'
</code></pre></div></div>
<p>After this setup, your VM will start every <em>Sunday</em>, backup all your photos of all accounts and shutdown on <em>Monday</em>.</p><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/NullOnError?a=OklUN9EQ-64:td-LpPFnRfs:yIl2AUoC8zA"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=yIl2AUoC8zA" /></a> <a href="http://feeds.feedburner.com/~ff/NullOnError?a=OklUN9EQ-64:td-LpPFnRfs:qj6IDK7rITs"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=qj6IDK7rITs" /></a>
</div><img alt="" height="1" src="http://feeds.feedburner.com/~r/NullOnError/~4/OklUN9EQ-64" width="1" />Why? Google Cloud Storage is cheaper, and you pay only for what you use than Google One. Also, you can erase any photo, and you still have a copy of that.Rodrigo Delducarodrigodelduca@gmail.comhttps://nullonerror.org/Auto generating SEO-friendly URLs with Scrapy pipelineshttps://nullonerror.org/2020/12/14/auto-generating-seo-friendly-urls-with-scrapy-pipelines2020-12-14T00:00:00+00:00<p>I was using Scrapy to crawl some websites and mirror their content into a new one and at the same time, generate beautiful and unique URLs based on the title, but the title can appear repeated! So I added part of the original URL in <a href="https://en.wikipedia.org/wiki/Base36">base36</a> as uniqueness guarantees.</p>
<p>In the <em>URL</em> I wanted the title without special symbols, only ASCII and at the end a unique and short inditifier, and part of the result of the <a href="https://en.wikipedia.org/wiki/SHA-2">SHA-256</a> of the URL in <em>base36</em>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">class</span> <span class="nc">PreparePipeline</span><span class="p">():</span>
<span class="k">def</span> <span class="nf">process_item</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">item</span><span class="p">,</span> <span class="n">spider</span><span class="p">):</span>
<span class="n">title</span> <span class="o">=</span> <span class="n">item</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="s">"title"</span><span class="p">)</span>
<span class="k">if</span> <span class="n">title</span> <span class="ow">is</span> <span class="bp">None</span><span class="p">:</span>
<span class="k">raise</span> <span class="n">DropItem</span><span class="p">(</span><span class="sa">f</span><span class="s">"No title were found on item: </span><span class="si">{</span><span class="n">item</span><span class="si">}</span><span class="s">."</span><span class="p">)</span>
<span class="n">url</span> <span class="o">=</span> <span class="n">item</span><span class="p">[</span><span class="s">"url"</span><span class="p">]</span>
<span class="n">N</span> <span class="o">=</span> <span class="mi">4</span>
<span class="n">sha256</span> <span class="o">=</span> <span class="n">hashlib</span><span class="p">.</span><span class="n">sha256</span><span class="p">(</span><span class="n">url</span><span class="p">.</span><span class="n">encode</span><span class="p">()).</span><span class="n">digest</span><span class="p">()</span>
<span class="n">sliced</span> <span class="o">=</span> <span class="nb">int</span><span class="p">.</span><span class="n">from_bytes</span><span class="p">(</span>
<span class="nb">memoryview</span><span class="p">(</span><span class="n">sha256</span><span class="p">)[:</span><span class="n">N</span><span class="p">].</span><span class="n">tobytes</span><span class="p">(),</span> <span class="n">byteorder</span><span class="o">=</span><span class="n">sys</span><span class="p">.</span><span class="n">byteorder</span><span class="p">)</span>
<span class="n">uid</span> <span class="o">=</span> <span class="n">base36</span><span class="p">.</span><span class="n">dumps</span><span class="p">(</span><span class="n">sliced</span><span class="p">)</span>
<span class="n">strip</span> <span class="o">=</span> <span class="nb">str</span><span class="p">.</span><span class="n">strip</span>
<span class="n">lower</span> <span class="o">=</span> <span class="nb">str</span><span class="p">.</span><span class="n">lower</span>
<span class="n">split</span> <span class="o">=</span> <span class="nb">str</span><span class="p">.</span><span class="n">split</span>
<span class="n">deunicode</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">normalize</span><span class="p">(</span><span class="s">"NFD"</span><span class="p">,</span> <span class="n">n</span><span class="p">).</span><span class="n">encode</span><span class="p">(</span><span class="s">"ascii"</span><span class="p">,</span> <span class="s">"ignore"</span><span class="p">).</span><span class="n">decode</span><span class="p">(</span><span class="s">"utf-8"</span><span class="p">)</span>
<span class="n">trashout</span> <span class="o">=</span> <span class="k">lambda</span> <span class="n">n</span><span class="p">:</span> <span class="n">re</span><span class="p">.</span><span class="n">sub</span><span class="p">(</span><span class="sa">r</span><span class="s">"[.,-@/\\|*]"</span><span class="p">,</span> <span class="s">" "</span><span class="p">,</span> <span class="n">n</span><span class="p">)</span>
<span class="n">functions</span> <span class="o">=</span> <span class="p">[</span><span class="n">strip</span><span class="p">,</span> <span class="n">deunicode</span><span class="p">,</span> <span class="n">trashout</span><span class="p">,</span> <span class="n">lower</span><span class="p">,</span> <span class="n">split</span><span class="p">]</span>
<span class="n">fragments</span> <span class="o">=</span> <span class="p">[</span>
<span class="o">*</span><span class="n">functools</span><span class="p">.</span><span class="nb">reduce</span><span class="p">(</span>
<span class="k">lambda</span> <span class="n">x</span><span class="p">,</span> <span class="n">f</span><span class="p">:</span> <span class="n">f</span><span class="p">(</span><span class="n">x</span><span class="p">),</span> <span class="n">functions</span><span class="p">,</span> <span class="n">title</span><span class="p">),</span>
<span class="n">uid</span><span class="p">,</span>
<span class="p">]</span>
<span class="n">item</span><span class="p">[</span><span class="s">"uid"</span><span class="p">]</span> <span class="o">=</span> <span class="s">"-"</span><span class="p">.</span><span class="n">join</span><span class="p">(</span><span class="n">fragments</span><span class="p">)</span>
<span class="k">return</span> <span class="n">item</span>
</code></pre></div></div>
<p>For example, with the <em>URL</em> <code class="language-plaintext highlighter-rouge">https://en.wikipedia.org/wiki/Déjà_vu</code> and <em>title</em> <code class="language-plaintext highlighter-rouge">Déjà vu - Wikipedia</code> will result in: <code class="language-plaintext highlighter-rouge">deja-vu-wikipedia-1q9i86k</code>. Which is perfect for my use case.</p><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/NullOnError?a=EDTqfZKMX08:5OAIzIQq2UI:yIl2AUoC8zA"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=yIl2AUoC8zA" /></a> <a href="http://feeds.feedburner.com/~ff/NullOnError?a=EDTqfZKMX08:5OAIzIQq2UI:qj6IDK7rITs"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=qj6IDK7rITs" /></a>
</div><img alt="" height="1" src="http://feeds.feedburner.com/~r/NullOnError/~4/EDTqfZKMX08" width="1" />I was using Scrapy to crawl some websites and mirror their content into a new one and at the same time, generate beautiful and unique URLs based on the title, but the title can appear repeated! So I added part of the original URL in base36 as uniqueness guarantees.Rodrigo Delducarodrigodelduca@gmail.comhttps://nullonerror.org/Taking advantage of Python’s concurrent futures to full saturate your bandwidthhttps://nullonerror.org/2020/12/13/full-saturate-your-bandwidth-with-python2020-12-13T00:00:00+00:00<blockquote>
<p>I am starting a new series of small snippets of code which I think that maybe useful or inspiring for others.</p>
</blockquote>
<p>Let’s suppose you have a pandas’ <code class="language-plaintext highlighter-rouge">dataframe</code> with a column named <em>URL</em> which one do you want to download.</p>
<p>The code below takes the advantage of the multi-core processing using the <a href="https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ThreadPoolExecutor">ThreadPoolExecutor</a> with <a href="https://requests.readthedocs.io/en/master/">requests</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">multiprocessing</span>
<span class="kn">import</span> <span class="nn">concurrent.futures</span>
<span class="kn">from</span> <span class="nn">requests</span> <span class="kn">import</span> <span class="n">Session</span>
<span class="kn">from</span> <span class="nn">requests.adapters</span> <span class="kn">import</span> <span class="n">HTTPAdapter</span>
<span class="kn">from</span> <span class="nn">urllib3.util</span> <span class="kn">import</span> <span class="n">Retry</span>
<span class="n">session</span> <span class="o">=</span> <span class="n">Session</span><span class="p">()</span>
<span class="n">retry</span> <span class="o">=</span> <span class="n">Retry</span><span class="p">(</span><span class="n">connect</span><span class="o">=</span><span class="mi">8</span><span class="p">,</span> <span class="n">backoff_factor</span><span class="o">=</span><span class="mf">0.5</span><span class="p">)</span>
<span class="n">adapter</span> <span class="o">=</span> <span class="n">HTTPAdapter</span><span class="p">(</span><span class="n">max_retries</span><span class="o">=</span><span class="n">retry</span><span class="p">)</span>
<span class="n">session</span><span class="p">.</span><span class="n">mount</span><span class="p">(</span><span class="s">"http://"</span><span class="p">,</span> <span class="n">adapter</span><span class="p">)</span>
<span class="n">session</span><span class="p">.</span><span class="n">mount</span><span class="p">(</span><span class="s">"https://"</span><span class="p">,</span> <span class="n">adapter</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">download</span><span class="p">(</span><span class="n">url</span><span class="p">):</span>
<span class="n">filename</span> <span class="o">=</span> <span class="s">"/"</span><span class="p">.</span><span class="n">join</span><span class="p">([</span><span class="s">"subdir"</span><span class="p">,</span> <span class="n">url</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">"/"</span><span class="p">)[</span><span class="o">-</span><span class="mi">1</span><span class="p">]])</span>
<span class="k">with</span> <span class="n">session</span><span class="p">.</span><span class="n">get</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">stream</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span> <span class="k">as</span> <span class="n">r</span><span class="p">:</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">r</span><span class="p">.</span><span class="n">ok</span><span class="p">:</span>
<span class="k">return</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">filename</span><span class="p">,</span> <span class="s">"wb"</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">r</span><span class="p">.</span><span class="n">iter_content</span><span class="p">(</span><span class="n">chunk_size</span><span class="o">=</span><span class="mi">8192</span><span class="p">):</span>
<span class="n">f</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="n">chunk</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">run</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">processes</span><span class="o">=</span><span class="n">multiprocessing</span><span class="p">.</span><span class="n">cpu_count</span><span class="p">()</span> <span class="o">*</span> <span class="mi">2</span><span class="p">):</span>
<span class="k">with</span> <span class="n">concurrent</span><span class="p">.</span><span class="n">futures</span><span class="p">.</span><span class="n">ThreadPoolExecutor</span><span class="p">(</span><span class="n">processes</span><span class="p">)</span> <span class="k">as</span> <span class="n">pool</span><span class="p">:</span>
<span class="nb">list</span><span class="p">(</span><span class="n">pool</span><span class="p">.</span><span class="nb">map</span><span class="p">(</span><span class="n">download</span><span class="p">,</span> <span class="n">df</span><span class="p">[</span><span class="s">"url"</span><span class="p">]))</span>
<span class="k">if</span> <span class="n">__name__</span> <span class="o">==</span> <span class="s">'__main__'</span><span class="p">:</span>
<span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="p">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s">"download.csv"</span><span class="p">)</span>
<span class="n">run</span><span class="p">(</span><span class="n">df</span><span class="p">)</span>
</code></pre></div></div><div class="feedflare">
<a href="http://feeds.feedburner.com/~ff/NullOnError?a=j_s5p8gL83Q:aeCgNQeP0Zo:yIl2AUoC8zA"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=yIl2AUoC8zA" /></a> <a href="http://feeds.feedburner.com/~ff/NullOnError?a=j_s5p8gL83Q:aeCgNQeP0Zo:qj6IDK7rITs"><img border="0" src="http://feeds.feedburner.com/~ff/NullOnError?d=qj6IDK7rITs" /></a>
</div><img alt="" height="1" src="http://feeds.feedburner.com/~r/NullOnError/~4/j_s5p8gL83Q" width="1" />I am starting a new series of small snippets of code which I think that maybe useful or inspiring for others.Rodrigo Delducarodrigodelduca@gmail.comhttps://nullonerror.org/Fazendo backup do banco de dados no Djangotag:pythonclub.com.br,2020-10-30:/fazendo-backup-do-banco-de-dados-no-django.html2020-10-30T13:40:00+00:00<h2>Apresentação</h2>
<p>Em algum momento, durante o seu processo de desenvolvimento com Django, pode ser que surja a necessidade de criar e restaurar o banco de dados da aplicação. Pensando nisso, resolvi fazer um pequeno tutorial, básico, de como realizar essa operação.</p>
<p>Nesse tutorial, usaremos o <a href="https://github.com/django-dbbackup/django-dbbackup">django-dbbackup</a>, um pacote desenvolvido especificamente para isso.</p>
<h2>Configurando nosso ambiente</h2>
<p>Primeiro, partindo do início, vamos criar uma pasta para o nosso projeto e, nela, isolar o nosso ambiente de desenvolvimento usando uma <a href="https://virtualenv.pypa.io/en/latest/index.html">virtualenv</a>:</p>
<div class="highlight"><pre><span></span>mkdir projeto_db <span class="o">&&</span> <span class="nb">cd</span> projeto_db <span class="c1">#criando a pasta do nosso projeto</span>
virtualenv -p python3.8 env <span class="o">&&</span> <span class="nb">source</span> env/bin/activate <span class="c1">#criando e ativando a nossa virtualenv</span>
</pre></div>
<p>Depois disso e com o nosso ambiente já ativo, vamos realizar os seguintes procedimentos:</p>
<div class="highlight"><pre><span></span>pip install -U pip <span class="c1">#com isso, atualizamos a verão do pip instalado</span>
</pre></div>
<h2>Instalando as dependências</h2>
<p>Agora, vamos instalar o <a href="https://www.djangoproject.com/">Django</a> e o pacote que usaremos para fazer nossos backups.</p>
<div class="highlight"><pre><span></span>pip install <span class="nv">Django</span><span class="o">==</span><span class="m">3</span>.1.2 <span class="c1">#instalando o Django</span>
pip install django-dbbackup <span class="c1">#instalando o django-dbbackup</span>
</pre></div>
<h2>Criando e configurando projeto</h2>
<p>Depois de instaladas nossas dependências, vamos criar o nosso projeto e configurar o nosso pacote nas configurações do Django.</p>
<div class="highlight"><pre><span></span>django-admin startproject django_db . <span class="c1">#dentro da nossa pasta projeto_db, criamos um projeto Django com o nome de django_db.</span>
</pre></div>
<p>Depois de criado nosso projeto, vamos criar e popular o nosso banco de dados.</p>
<div class="highlight"><pre><span></span>python manage.py migrate <span class="c1">#com isso, sincronizamos o estado do banco de dados com o conjunto atual de modelos e migrações.</span>
</pre></div>
<p>Criado nosso banco de dados, vamos criar um superusuário para podemos o painel admin do nosso projeto.</p>
<div class="highlight"><pre><span></span>python manage.py createsuperuser
</pre></div>
<p>Perfeito. Já temos tudo que precisamos para executar nosso projeto. Para execução dele, é só fazermos:</p>
<div class="highlight"><pre><span></span>python manage.py runserver
</pre></div>
<p>Você terá uma imagem assim do seu projeto:</p>
<p><img alt="" src="https://jacksonosvaldo.github.io/img/django_db.png" /></p>
<h2>Configurando o django-dbbackup</h2>
<p>Dentro do seu projeto, vamos acessar o arquivo settings.py, como expresso abaixo:</p>
<div class="highlight"><pre><span></span>django_db/
├── settings.py
</pre></div>
<p>Dentro desse arquivos iremos, primeiro, adiconar o django-dbbackup às apps do projeto:</p>
<div class="highlight"><pre><span></span><span class="n">INSTALLED_APPS</span> <span class="o">=</span> <span class="p">(</span>
<span class="o">...</span>
<span class="s1">'dbbackup'</span><span class="p">,</span> <span class="c1"># adicionando django-dbbackup</span>
<span class="p">)</span>
</pre></div>
<p>Depois de adicionado às apps, vamos dizer para o Django o que vamos salvar no backup e, depois, indicar a pasta para onde será encaminhado esse arquivo. Essa inserção deve ou pode ser feita no final do arquivo <em>settings.py</em>:</p>
<div class="highlight"><pre><span></span><span class="n">DBBACKUP_STORAGE</span> <span class="o">=</span> <span class="s1">'django.core.files.storage.FileSystemStorage'</span> <span class="c1">#o que salvar</span>
<span class="n">DBBACKUP_STORAGE_OPTIONS</span> <span class="o">=</span> <span class="p">{</span><span class="s1">'location'</span><span class="p">:</span> <span class="s1">'backups/'</span><span class="p">}</span> <span class="c1"># onde salvar</span>
</pre></div>
<p>Percebam que dissemos para o Django salvar o backup na pasta <em>backups</em>, mas essa pasta ainda não existe no nosso projeto. Por isso, precisamos criá-la [fora da pasta do projeto]:</p>
<div class="highlight"><pre><span></span>mkdir backups
</pre></div>
<h2>Criando e restaurando nosso backup</h2>
<p>Já temos tudo pronto. Agora, vamos criar o nosso primeiro backup:</p>
<div class="highlight"><pre><span></span>python manage.py dbbackup
</pre></div>
<p>Depois de exetudado, será criado um arquivo -- no nosso exemplo, esse arquivo terá uma extensão .dump --, salvo na pasta <em>backups</em>. Esse arquivo contem todo backup do nosso banco de dados.</p>
<p>Para recuperarmos nosso banco, vamos supor que migramos nosso sistema de um servidor antigo para um novo e, por algum motivo, nossa base de dados foi corrompida, inviabilizando seu uso. Ou seja, estamos com o sistema/projeto sem banco de dados -- ou seja, exlua ou mova a a sua base dados .sqlite3 para que esse exemplo seja útil --, mas temos os backups. Com isso, vamos restaurar o banco:</p>
<div class="highlight"><pre><span></span>python manage.py dbrestore
</pre></div>
<p>Prontinho, restauramos nosso banco de dados. O interessante do django-dbbackup, dentre outras coisas, é que ele gera os backups com datas e horários específicos, facilitando o processo de recuperação das informações mais recentes.</p>
<p>Por hoje é isso, pessoal. Até a próxima. ;) </p>Jackson Osvaldohttp://pythonclub.com.br/Seqtembro de eventos virtuais e gratuitos sobre Qt e KDEhttps://blog.filipesaraiva.info/?p=21852020-08-29T18:48:00+00:00<p>(Ok a piada com <em>seqtembro</em> funciona melhor na versão em inglês, <em>seqtember</em>, mas simbora)</p>
<p>Por uma grande coincidência, obra do destino, ou nada disso, teremos um Setembro de 2020 repleto de eventos virtuais e gratuitos de alta qualidade sobre <a href="https://www.qt.io/" rel="noopener noreferrer" target="_blank">Qt</a> e <a href="https://kde.org/" rel="noopener noreferrer" target="_blank">KDE</a>.</p>
<p>Começando de 4 à 11 do referido mês teremos o <a href="https://akademy.kde.org/2020/" rel="noopener noreferrer" target="_blank">Akademy 2020</a>, o grande encontro mundial da comunidade KDE que esse ano, por motivos que todos sabemos, acontecerá de forma virtual. A <a href="https://akademy.kde.org/2020/program" rel="noopener noreferrer" target="_blank">programação</a> do Akademy traz palestras, treinamentos, <em>hacking sessions</em>, discussões com foco em aplicações KDE específicas, e mais, reunindo hackers, designers, gerentes de projetos, tradutores, e colabores dos mais diversos segmentos para discutir e planejar o KDE e seus futuros passos.</p>
<p><a href="https://blog.filipesaraiva.info/?attachment_id=2190" rel="attachment wp-att-2190"><img alt="" class="aligncenter wp-image-2190" height="260" src="https://blog.filipesaraiva.info/wp-content/uploads/2020/08/collage_logo-1024x346.jpg" width="770" /></a></p>
<p>E como falamos em KDE, por extensão, também falamos em Qt – afinal, grande parte das aplicações é escrita nesse framework. Portanto, mesmo que você trabalhe com Qt mas não use nada do KDE, vale a pena participar do evento – e também se perguntar “porque diabos não estou usando e desenvolvendo aplicações do KDE?”.</p>
<p>Um incentivo extra é que durante o Akademy, entre 7 e 11, acontecerá o <a href="https://www.qtdesktopdays.com/" rel="noopener noreferrer" target="_blank">Qt Desktop Days</a>, evento da <a href="https://www.kdab.com/" rel="noopener noreferrer" target="_blank">KDAB</a> voltado para Qt no desktop (surpresa?). A <a href="https://www.qtdesktopdays.com/program/" rel="noopener noreferrer" target="_blank">programação preliminar</a> já está disponível e será muito interessante ver os avanços da tecnologia em um campo que pode parecer menos <em>sexy</em> hoje em dia, por conta da muita atenção dada a projetos mobile ou embarcados, mas que pelo contrário, continua vibrante e recebendo muito investimento.</p>
<p><img alt="" class="aligncenter wp-image-2191" height="520" src="https://blog.filipesaraiva.info/wp-content/uploads/2020/08/IMG_20200829_154545_039-1024x1024.jpg" width="520" /></p>
<p>Após uma rápida pausa para um respiro, temos a <a href="http://bit.do/maratonaqt" rel="noopener noreferrer" target="_blank">Maratona Qt</a>. Nosso amigo <a href="https://sandroandrade.org/" rel="noopener noreferrer" target="_blank">Sandro Andrade</a>, professor do IFBA e colaborador de longa data do KDE, resolveu dedicar uma semana inteira, de 14 à 18 de setembro, para apresentar 5 tópicos sobre o Qt tratando de seus fundamentos e passos iniciais de cada um. O programa cobre QML, C++ e Qt, Qt no Android, no iOS, na web (sim!), computação gráfica e mesmo jogos! Extremamente recomendada pra todo mundo que conhece ou quer conhecer o framework.</p>
<p>A Maratona Qt vai servir como um esquenta para a <a href="https://br.qtcon.org/" rel="noopener noreferrer" target="_blank">QtCon Brasil 2020</a>, esse ano também virtual. Em 26 e 27 de setembro o pessoal da <a href="http://qmob.solutions/" rel="noopener noreferrer" target="_blank">qmob.solutions</a> reunirá desenvolvedores Qt de vários países para apresentarem, entre outras coisas, trabalhos com Wayland, visão computacional e IA, análise de dados, Python, containers, prototipagem, embarcados, e outros, tudo envolvendo Qt! E também haverá uma apresentação sobre a próxima versão <em>major</em> da ferramenta, Qt 6.</p>
<p>Portanto pessoal, reservem este mês para uma grande imersão nos vários aspectos e possibilidades disponibilizadas pelo Qt.</p>Filipe Saraivahttps://blog.filipesaraiva.infoKubicast - Episódio 45: Arquitetura de Software, existe algo além dos microsserviços?https://avelino.run/kubicast-epis%C3%B3dio-45-arquitetura-de-software-existe-algo-al%C3%A9m-dos-microsservi%C3%A7os/2020-08-27T00:00:00+00:00Escute minha participação no Kubicast junto com Felipe Oliveira falando sobre Arquitetura de Micros Serviços em comparação com Monolito:<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/bmpARbEKUfY" width="1" />Thiago Avelinohttps://avelino.run/blog/MinHashing all the things: a quick analysis of MAG search resultstag:blog.luizirber.org,2020-07-24:/2020/07/24/mag-results/2020-07-24T15:00:00+00:00<p><a href="https://blog.luizirber.org/2020/07/22/mag-search/">Last time</a> I described a way to search MAGs in metagenomes,
and teased about interesting results.
Let's dig in some of them!</p>
<p>I prepared a <a href="https://github.com/luizirber/2020-07-22-mag-search/">repo</a> with the data and a notebook with the analysis I did in this
post.
You can also follow along in <a href="https://mybinder.org">Binder</a>,
as well as do your own analysis! <a href="https://mybinder.org/v2/gh/luizirber/2020-07-22-mag-search/master?filepath=index.ipynb"><img alt="Binder" src="https://mybinder.org/badge_logo.svg" /></a></p>
<h2>Preparing some metadata</h2>
<p>The supplemental materials for <a href="https://www.nature.com/articles/sdata2017203">Tully et al</a> include more details about each MAG,
so let's download them.
I prepared a small snakemake workflow to do that,
as well as downloading information about the SRA datasets from Tara Oceans
(the dataset used to generate the MAGs),
as well as from <a href="https://www.nature.com/articles/s41564-017-0012-7">Parks et al</a>,
which also generated MAGs from Tara Oceans.
Feel free to include them in your analysis,
but I was curious to find matches in other metagenomes.</p>
<h2>Loading the data</h2>
<p>The results from the MAG search are in a CSV file,
with a column for the MAG name,
another for the SRA dataset ID for the metagenome and a third column for the
containment of the MAG in the metagenome.
I also fixed the names to make it easier to query,
and finally removed the Tara and Parks metagenomes
(because we already knew they contained these MAGs).</p>
<p>This left us with 23,644 SRA metagenomes with matches,
covering 2,291 of the 2,631 MAGs.
These are results for a fairly low containment (10%),
so if we limit to MAGs with more than 50% containment we still have 1,407 MAGs and 2,938 metagenomes left.</p>
<h2>TOBG_NP-110, I choose you!</h2>
<p>That's still a lot,
so I decided to pick a candidate to check before doing any large scale analysis.
I chose TOBG_NP-110 because there were many matches above 50% containment,
and even some at 99%.
Turns out it is also an Archaeal MAG that failed to be classified further than Phylum level (Euryarchaeota),
with a 70.3% complete score in the original analysis.
Oh, let me dissect the name a bit:
TOBG is "Tara Ocean Binned Genome" and "NP" is North Pacific.</p>
<p>And so I went checking where the other metagenome matches came from.
5 of the 12 matches above 50% containment come from one study,
<a href="https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP044185">SRP044185</a>,
with samples collected from a column of water in a station in Manzanillo, Mexico.
Other 3 matches come from
<a href="https://trace.ncbi.nlm.nih.gov/Traces/sra/?study=SRP003331">SRP003331</a>,
in the South Pacific ocean (in northern Chile).
Another match,
<a href="https://trace.ncbi.nlm.nih.gov/Traces/sra/?run=ERR3256923">ERR3256923</a>,
also comes from the South Pacific.</p>
<h2>What else can I do?</h2>
<p>I'm curious to follow <a href="http://merenlab.org/data/refining-mags/">the refining MAGs</a> tutorial from the Meren Lab and see where this goes,
and especially in using <a href="https://genomebiology.biomedcentral.com/articles/10.1186/s13059-020-02066-4"><code>spacegraphcats</code></a>
to extract neighborhoods from the MAG and better evaluate what is missing or if there are other interesting bits that
the MAG generation methods ended up discarding.</p>
<p>So, for now that's it.
But more important,
I didn't want to sit on these results until there is a publication in press,
especially when there are people that can do so much more with these,
so I decided to make it all public.
It is way more exciting to see this being used to know more about these
organisms than me being the only one with access to this info.</p>
<p>And yesterday I saw <a href="https://twitter.com/DrJonathanRosa/status/1286381346605027328">this tweet</a> by
<a href="https://twitter.com/DrJonathanRosa/status/1286381346605027328">@DrJonathanRosa</a>,
saying:</p>
<blockquote>
<p>I don’t know who told students that the goal of research is to find some
previously undiscovered research topic, claim individual ownership over it,
& fiercely protect it from theft, but that almost sounds like, well,
colonialism, capitalism, & policing </p>
</blockquote>
<p>Amen.</p>
<h2>I want to run this with my data!</h2>
<p>Next time. But we will have a discussion about scientific infrastructure and
sustainability first =]</p>
<h2>Comments?</h2>
<ul>
<li><a href="https://twitter.com/luizirber/status/1286700888111738880">Thread on Twitter</a></li>
</ul>luizirberhttps://blog.luizirber.org/Comunicação é a base dos projetos Open Sourcehttps://avelino.run/comunica%C3%A7%C3%A3o-%C3%A9-a-base-dos-projetos-open-source/2020-07-23T00:00:00+00:00Sou criador e mantenedor (junto com uma incrivel comunidade, composta por pessoas ao redor do mundo) de um projeto chamado awesome-go Lista curada pela comunidade de frameworks, bibliotecas e software escritos em Go.
Quando comecei contribuir e criar projeto Open Source achava que o foco principal era código, com o passar dos anos comecei perceber que o projeto era um meio para chegar em algum lugar, ou seja, código tem sua importância, mas não basta tem um projeto com código impecável vendo que “ninguém” ou poucas pessoas conseguem usar.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/Mojch08DxKw" width="1" />Thiago Avelinohttps://avelino.run/blog/MinHashing all the things: searching for MAGs in the SRAtag:blog.luizirber.org,2020-07-22:/2020/07/22/mag-search/2020-07-22T15:00:00+00:00<p>(or: Top-down and bottom-up approaches for working around sourmash limitations)</p>
<p>In the last month I updated <a href="https://wort.oxli.org">wort</a>,
the system I developed for computing sourmash signature for public genomic databases,
and started calculating signatures
for the <a href="https://www.ncbi.nlm.nih.gov/sra/?term=%22METAGENOMIC%22%5Bsource%5D+NOT+amplicon%5BAll+Fields%5D)">metagenomes</a> in the <a href="https://www.ncbi.nlm.nih.gov/sra/">Sequence Read Archive</a>.
This is a more challenging subset than the <a href="https://blog.luizirber.org/2016/12/28/soursigs-arch-1/">microbial datasets</a> I was doing previously,
since there are around 534k datasets from metagenomic sources in the SRA,
totalling 447 TB of data.
Another problem is the size of the datasets,
ranging from a couple of MB to 170 GB.
Turns out that the workers I have in <code>wort</code> are very good for small-ish datasets,
but I still need to figure out how to pull large datasets faster from the SRA,
because the large ones take forever to process...</p>
<p>The good news is that I managed to calculate signatures for almost 402k of them
<sup id="sf-mag-search-1-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-mag-search-1" title="pulling about a 100 TB in 3 days, which was pretty fun to see because I ended up DDoS myself because I couldn't download the generated sigs fast enough from the S3 bucket where they are temporarily stored =P">1</a></sup>,
which already let us work on some pretty exciting problems =]</p>
<h2>Looking for MAGs in the SRA</h2>
<p>Metagenome-assembled genomes are essential for studying organisms that are hard to isolate and culture in lab,
especially for environmental metagenomes.
<a href="https://www.nature.com/articles/sdata2017203">Tully et al</a> published 2,631 draft MAGs from 234 samples collected during the Tara Oceans expedition,
and I wanted to check if they can also be found in other metagenomes besides the Tara Oceans ones.
The idea is to extract the reads from these other matches and evaluate how the MAG can be improved,
or at least evaluate what is missing in them.
I choose to use environmental samples under the assumption they are easier to deposit on the SRA and have public access,
but there are many human gut microbiomes in the SRA and this MAG search would work just fine with those too.</p>
<p>Moreover,
I want to search for containment,
and not similarity.
The distinction is subtle,
but similarity takes into account both datasets sizes
(well, the size of the union of all elements in both datasets),
while containment only considers the size of the query.
This is relevant because the similarity of a MAG and a metagenome is going to be very small (and is symmetrical),
but the containment of the MAG in the metagenome might be large
(and is asymmetrical, since the containment of the metagenome in the MAG is likely very small because the metagenome is so much larger than the MAG).</p>
<h2>The computational challenge: indexing and searching</h2>
<p>sourmash signatures are a small fraction of the original size of the datasets,
but when you have hundreds of thousands of them the collection ends up being pretty large too.
More precisely, 825 GB large.
That is way bigger than any index I ever built for sourmash,
and it would also have pretty distinct characteristics than what we usually do:
we tend to index genomes and run <code>search</code> (to find similar genomes) or <code>gather</code>
(to decompose metagenomes into their constituent genomes),
but for this MAG search I want to find which metagenomes have my MAG query above a certain containment threshold.
Sort of a <code>sourmash search --containment</code>,
but over thousands of metagenome signatures.
The main benefit of an SBT index in this context is to avoid checking all signatures because we can prune the search early,
but currently SBT indices need to be totally loaded in memory during <code>sourmash index</code>.
I will have to do this in the medium term,
but I want a solution NOW! =]</p>
<p><a href="https://github.com/dib-lab/sourmash/releases/tag/v3.4.0">sourmash 3.4.0</a> introduced <code>--from-file</code> in many commands,
and since I can't build an index I decided to use it to load signatures for the metagenomes.
But... <code>sourmash search</code> tries to load all signatures in memory,
and while I might be able to find a cluster machine with hundreds of GBs of RAM available,
that's not very practical.</p>
<p>So, what to do?</p>
<h2>The top-down solution: a snakemake workflow</h2>
<p>I don't want to modify sourmash now,
so why not make a workflow and use snakemake to run one <code>sourmash search --containment</code> for each metagenome?
That means 402k tasks,
but at least I can use <a href="https://snakemake.readthedocs.io/en/stable/executing/cli.html#dealing-with-very-large-workflows">batches</a> and <a href="https://slurm.schedmd.com/job_array.html">SLURM job arrays</a> to submit reasonably-sized jobs to our HPC queue.
After running all batches I summarized results for each task,
and it worked well for a proof of concept.</p>
<p>But... it was still pretty resource intensive:
each task was running one query MAG against one metagenome,
and so each task needed to do all the overhead of starting the Python interpreter and parsing the query signature,
which is exactly the same for all tasks.
Extending it to support multiple queries to the same metagenome would involve duplicating tasks,
and 402k metagenomes times 2,631 MAGs is...
a very large number of jobs.</p>
<p>I also wanted to avoid clogging the job queues,
which is not very nice to the other researchers using the cluster.
This limited how many batches I could run in parallel...</p>
<h2>The bottom-up solution: Rust to the rescue!</h2>
<p>Thinking a bit more about the problem,
here is another solution:
what if we load all the MAGs in memory
(as they will be queried frequently and are not that large),
and then for each metagenome signature load it,
perform all MAG queries,
and then unload the metagenome signature from memory?
This way we can control memory consumption
(it's going to be proportional to all the MAG sizes plus the size of the largest metagenome)
and can also efficiently parallelize the code because each task/metagenome is independent
and the MAG signatures can be shared freely (since they are read-only).</p>
<p>This could be done with the sourmash Python API plus <code>multiprocessing</code> or some
other parallelization approach (maybe dask?),
but turns out that everything we need comes from the Rust API.
Why not enjoy a bit of the <a href="https://doc.rust-lang.org/stable/book/ch16-00-concurrency.html">fearless concurrency</a> that is one of the major Rust goals?</p>
<p><a href="https://github.com/luizirber/phd/blob/aa1ed9eb33ba71fdf9b3f2c92931701be6df00cd/experiments/wort/sra_search/src/main.rs">The whole code</a> ended up being 176 lines long,
including command line parsing using <a href="https://docs.rs/structopt/latest/structopt/">strucopt</a> and parallelizing the search using <a href="https://docs.rs/rayon/latest/rayon/">rayon</a>
and a <a href="https://doc.rust-lang.org/std/sync/mpsc/fn.channel.html">multiple-producer, single-consumer channel</a> to write results to an output
(either the terminal or a file).
This version took 11 hours to run,
using less than 5GB of RAM and 32 processors,
to search 2k MAGs against 402k metagenomes.
And, bonus! It can also be parallelized again if you have multiple machines,
so it potentially takes a bit more than an hour to run if you can allocate 10 batch jobs,
with each batch 1/10 of the metagenome signatures.</p>
<h2>So, is bottom-up always the better choice?</h2>
<p>I would like to answer "Yes!",
but bioinformatics software tends to be organized as command line interfaces,
not as libraries.
Libraries also tend to have even less documentation than CLIs,
and this particular case is not a fair comparison because...
Well, I wrote most of the library,
and the Rust API is not that well documented for general use.</p>
<p>But I'm pretty happy with how the sourmash CLI is viable both for the top-down approach
(and whatever workflow software you want to use) as well as how the Rust core worked for the bottom-up approach.
I think the most important is having the option to choose which way to go,
especially because now I can use the bottom-up approach to make the sourmash CLI
and Python API better.
The top-down approach is also way more accessible in general,
because you can pick your favorite workflow software and use all the tricks you're comfortable with.</p>
<h2>But, what about the results?!?!?!</h2>
<p>Next time. But I did find MAGs with over 90% containment in very different locations,
which is pretty exciting!</p>
<p>I also need to find a better way of distributing all these signature,
because storing 4 TB of data in S3 is somewhat cheap,
but transferring data is very expensive.
All signatures are also available on IPFS,
but I need more people to host them and share.
Get in contact if you're interested in helping =]</p>
<p>And while I'm asking for help,
any tips on pulling data faster from the SRA are greatly appreciated!</p>
<h2>Comments?</h2>
<ul>
<li><a href="https://twitter.com/luizirber/status/1285782732790849537">Thread on Twitter</a></li>
</ul><hr /><h2>Footnotes</h2><ol><li id="sf-mag-search-1"><p>pulling about a 100 TB in 3 days, which was pretty fun to see because I
ended up DDoS myself because I couldn't download the generated sigs fast enough
from the S3 bucket where they are temporarily stored =P <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-mag-search-1-back">↩</a></p></li></ol>luizirberhttps://blog.luizirber.org/Por trás de projetos Open Source existe pessoashttps://avelino.run/por-tr%C3%A1s-de-projetos-open-source-existe-pessoas/2020-07-08T00:00:00+00:00Tecnologia, sejá humano ao receber contribuição"
Muitos engenheiro(a)s esquecem ao contribuir com projetos Open Source que por trás de todos projetos temos pessoas.
Não conhecemos as pessoas que estão do outro lado (mantenedores do projeto) e como eles receberá nossa contribuição, isso nos gera a necessidade da comunicação ser extremamente clara e não assumirmos que os mantenedores (contribuidores) tenha o mesmo conhecimento que nós (não temos como saber o que as outras pessoas tem de conhecimento), mesmo conceitos que achamos óbvios é importante deixar claro na comunicação (issue, pull request e etc).<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/MlsNaE1pTos" width="1" />Thiago Avelinohttps://avelino.run/blog/Efeitos gráficos com Python, Tkinter, Cython e Numbahttps://blog.nilo.pro.br/posts/2020-05-24-efeitos-graficos-com-python-cython-e-numba/2020-05-24T07:25:00+00:00<p>Ontem, sábado, fiquei com vontade de criar um efeito de flamas (fogo) em Python. Este efeito era bem popular no início dos anos 90. Eu lembrava que o algoritmo era bem simples, mas tinha uns truques a fazer com a paleta de cores.</p>
<p>Achei este artigo com a implementação em C: <a href="https://lodev.org/cgtutor/fire.html">https://lodev.org/cgtutor/fire.html</a></p>
<p>Do mesmo artigo, podemos ter uma ideia de como fica o efeito:</p>
<p><img alt="Efeito de flamas" src="https://lodev.org/cgtutor/images/fire.jpg" title="Efeito de flamas" /></p>
<p>Depois de ler o artigo e ver uns vídeos no Youtube, vi-me com dois problemas:</p>
<ol>
<li>
<p>Precisava de uma aplicação gráfica capaz de mostrar imagens, como uma animação, uma imagem após a outra, o mais rápido possível (no mínimo uns 15 frames por segundo, idealmente acima de 30).</p>
</li>
<li>
<p>Suspeitei que teria problemas de velocidade para gerar as imagens, uma vez que uma mera imagem de 1024 x 1024 tem muitos pontos e usa uns 3 bytes por ponto. Imaginando uma matriz deste tamanho para trabalhar em Python, vi que não seria tão fácil escrever esta parte apenas em Python. Instalei o numpy para garantir.</p>
</li>
</ol>
<p>Eu esperava que o problema um seria relativamente simples, mas já explico o que complicou um pouco. Como eu quero apenas mostrar uma imagem, o tkinter do Python já seria suficiente. Comecei por criar uma aplicação simples, mostrando um Canvas e adicionando uma imagem. Porém, devido ao problema dois, durante o tempo para gerar a imagem, a tela fica completamente bloqueada, você não consegue mover ou fechar a janela.</p>
<p>O código está em português e em inglês, mas basicamente é uma aplicação tkinter, onde a janela principal tem um Label para apresentar uma mensagem, no caso, o número do frame corrente; e uma imagem.</p>
<div class="highlight"><pre><code class="language-python"><span>class</span> <span>App</span>(tk<span>.</span>Tk):
<span>def</span> __init__(self, desenhador, func, preFunc, <span>*</span>args, <span>**</span>kwargs):
super()<span>.</span>__init__(<span>*</span>args, <span>**</span>kwargs)
self<span>.</span>setup_windows()
self<span>.</span>queue <span>=</span> Queue()
self<span>.</span>queueStop <span>=</span> Queue()
self<span>.</span>setup_thread(desenhador, func, preFunc)
self<span>.</span>buffer <span>=</span> None
self<span>.</span>running <span>=</span> True
self<span>.</span>dead <span>=</span> False
self<span>.</span>after(<span>1</span>, self<span>.</span>check_queue)
<span>def</span> <span>setup_windows</span>(self):
self<span>.</span>title(<span>'Gerador de Imagens'</span>)
self<span>.</span>status <span>=</span> tk<span>.</span>StringVar(self, value<span>=</span><span>'Aguardando'</span>)
tk<span>.</span>Label(self, textvariable<span>=</span>self<span>.</span>status)<span>.</span>pack()
self<span>.</span>canvas <span>=</span> tk<span>.</span>Canvas(self, width<span>=</span>LARGURA, height<span>=</span>ALTURA)
self<span>.</span>image <span>=</span> self<span>.</span>canvas<span>.</span>create_image(<span>0</span>, <span>0</span>, anchor<span>=</span>tk<span>.</span>NW)
self<span>.</span>canvas<span>.</span>pack()
self<span>.</span>protocol(<span>"WM_DELETE_WINDOW"</span>, self<span>.</span>terminate)
</code></pre></div><p>O método <code>setup_windows</code> configura a janela, adicionando o Label, criando o Canvas e a imagem. Como vamos trocar a imagem frequentemente, ela também guarda uma referência a imagem no canvas em <code>self.image</code>. Este método também configura a janela para chamar <code>self.terminate</code> caso o usuário a feche.</p>
<p>Já <code>setup_thread</code> faz a inicialização do thread, a classe que gerencia o thread é passada como desenhador ao <code>__init__</code>. Para facilitar a comunicação com o thread, duas filas foram criadas, uma para receber as mensagens vindas do thread (<code>self.queue</code>) e outra para esperar a finalização do thread <code>self.queueStop</code> (mais detalhes depois). <code>func</code> e <code>preFunc</code> são duas funções usadas para facilitar os testes, onde as funções que realizam o desenho da imagem podem ser passadas como parâmetro. <code>preFunc</code> gera a primeira imagem e <code>func</code> é chamada dentro de um loop para gerar as imagens (frames seguintes). O thread desenhador é iniciado imediatamente após sua criação.</p>
<div class="highlight"><pre><code class="language-python"> <span>def</span> <span>setup_thread</span>(self, desenhador, func, preFunc):
self<span>.</span>desenhador <span>=</span> desenhador(self<span>.</span>queue, self<span>.</span>queueStop, func, preFunc)
self<span>.</span>desenhador<span>.</span>start()
</code></pre></div><p>Uma vez que a janela e o thread que atualiza as imagens já foram criados, precisamos de um método que fique periodicamente verificando se há novas imagens na fila. Este método é o <code>check_queue</code>, chamado no <code>__init__</code> com <code>self.after(1, self.check_queue)</code>. O uso de <code>self.after</code> é capital, pois começa a executar <code>check_queue</code> fora do <code>__init__</code>, depois da criação da janela e do loop de eventos.</p>
<p><code>check_queue</code> verifica se a fila com as imagens geradas pelo thread está vazia. Caso esteja, não faz nada, mas caso contrário, pega a nova imagem e troca a imagem do Canvas. No final, se agenda para rodar de novo 10 ms depois e repete este processo para trocar as imagens o quanto antes.</p>
<div class="highlight"><pre><code class="language-python"> <span>def</span> <span>check_queue</span>(self):
<span>if</span> <span>not</span> self<span>.</span>queue<span>.</span>empty():
contador, self<span>.</span>buffer <span>=</span> self<span>.</span>queue<span>.</span>get()
self<span>.</span>status<span>.</span>set(f<span>"Frame: {contador}"</span>)
self<span>.</span>canvas<span>.</span>itemconfig(self<span>.</span>image, image<span>=</span>self<span>.</span>buffer)
self<span>.</span>queue<span>.</span>task_done()
<span>if</span> self<span>.</span>running:
self<span>.</span>after(<span>10</span>, self<span>.</span>check_queue)
</code></pre></div><p>Quando trabalhamos com múltiplos threads no tkinter e na maioria dos frameworks GUI, normalmente, só podemos alterar os objetos geridos pelo framework no mesmo thread que roda o <code>mainloop</code>.
É por isso que a imagem é trocada em <code>check_queue</code>. Isto também leva a outros problemas a gerir entre os threads e a GUI. Por exemplo, a conversão de uma imagem, realizada no thread que desenha (detalhes depois), precisa que o tkinter esteja rodando e processando eventos, mesmo sendo um objeto fora da tela e não associado a nenhum controle. Esta é uma característica do tkinter. E por isto, <code>terminate</code> chama <code>check_thread_dead</code> para matar o thread, mas esperando como o loop principal da tkinter rodando. Veja que o desenhador é parado com <code>self.desenhador.stop()</code>. Depois, <code>check_thread_dead</code> é chamada para verificar se o desenhador realmente parou, é neste momento que usamos a outra fila, <code>queueStop</code>. Esta fila fica vazia durante a execução do programa e só recebe algo quando o loop do desenhador termina seu trabalho. Só então que o loop da tkinter é destruído com a chamada a <code>self.destroy()</code>.</p>
<div class="highlight"><pre><code class="language-python"> <span>def</span> <span>check_thread_dead</span>(self):
<span>if</span> self<span>.</span>queueStop<span>.</span>empty() <span>and</span> <span>not</span> self<span>.</span>dead:
self<span>.</span>after(<span>1</span>, self<span>.</span>check_thread_dead)
<span>return</span>
self<span>.</span>queueStop<span>.</span>get()
self<span>.</span>dead <span>=</span> True
self<span>.</span>desenhador<span>.</span>join()
self<span>.</span>destroy()
<span>def</span> <span>terminate</span>(self, e<span>=</span>None):
self<span>.</span>running <span>=</span> False
<span>if</span> <span>not</span> self<span>.</span>dead:
self<span>.</span>desenhador<span>.</span>stop()
self<span>.</span>check_thread_dead()
</code></pre></div><p>Isso tudo apenas para ter a janela sendo atualizada por outro thread. Ainda não desenhamos nada. Vejamos uma implementação de desenhador:</p>
<div class="highlight"><pre><code class="language-python"><span>class</span> <span>Desenha</span>(Thread):
<span>def</span> __init__(self, queue, queueStop, func, preFunc, <span>*</span>args, <span>**</span>kwargs):
super()<span>.</span>__init__(<span>*</span>args, <span>**</span>kwargs)
self<span>.</span>queue <span>=</span> queue
self<span>.</span>queueStop <span>=</span> queueStop
self<span>.</span>running <span>=</span> True
self<span>.</span>func <span>=</span> func
self<span>.</span>preFunc <span>=</span> preFunc <span>or</span> func
<span>def</span> <span>run</span>(self):
<span>try</span>:
data <span>=</span> numpy<span>.</span>zeros((ALTURA, LARGURA, <span>3</span>), dtype<span>=</span>numpy<span>.</span>uint8)
c <span>=</span> <span>0</span>
self<span>.</span>preFunc(data, c, LARGURA, ALTURA)
<span>while</span> self<span>.</span>running:
<span>with</span> TimeIt(<span>"Loop"</span>) <span>as</span> t:
<span># with TimeIt("ForLoop") as t:</span>
self<span>.</span>func(data, c, LARGURA, ALTURA)
<span># with TimeIt("FROM ARRAY") as t1:</span>
image <span>=</span> Image<span>.</span>fromarray(data)
<span># with TimeIt("Convert") as t2:</span>
converted_image <span>=</span> ImageTk<span>.</span>PhotoImage(image)
<span># with TimeIt("Queue") as t3:</span>
self<span>.</span>queue<span>.</span>put((c, converted_image))
c <span>+=</span> <span>1</span>
<span>finally</span>:
self<span>.</span>running <span>=</span> False
self<span>.</span>queueStop<span>.</span>put((<span>0</span>, <span>'FEITO'</span>))
<span>def</span> <span>stop</span>(self):
self<span>.</span>running <span>=</span> False
</code></pre></div><p>A classe <code>Desenha</code> recebe as filas para onde vai enviar as imagens que serão criadas dentro do <code>run</code> e também a mensagem que indica ter terminado. O trabalho em si é realizado dentro de <code>run</code> que é executado quando o thread é iniciado.</p>
<p>Como as matrizes são grandes, com facilmente mais de 1 milhão de elementos para imagens 1024 x 1024 pontos, <code>Desenha</code> utiliza arrays otimizados da numpy. Seria simplesmente muito mais lento trabalhar com listas do python para realizar estas operações, pois temos que preencher todos os pontos a cada imagem.</p>
<p>Se você não conhece a NumPy, ela é uma biblioteca muito utilizada em ciência de dados e várias outras áreas que precisam realizar operações com matrizes e efetuar cálculos matemáticos em geral em Python. Voce pode ler a documentação aqui <a href="https://numpy.org/doc/stable/reference/">NumPy</a>. A grande vantagem da NumPy é ser otimizada em C, além de fazer parte do Scipy.org.</p>
<p>Voltando ao <code>run</code>, ele basicamente cria uma matriz com tamanho suficiente para representar os pontos da nova imagem que vamos criar. Estas dimensões são ALTURA e LARGURA em 3 dimensões, uma para cada componente de cor RGB (Vermelho, Verde e Azul; um byte para cada). Assim, com uma imagem de 1024 x 1024 pontos, temos 1024 x 1024 x 3 = 3.145.728 bytes só para armazenar a matriz de pontos.</p>
<p>Uma vez que a matriz é criada, <code>run</code> chama a função de desenho <code>self.preFunc</code>, que realiza o desenho da primeira imagem, passando a matriz, um contador de frames, assim como as dimensões da imagem. Esta assinatura foi se desenvolvendo conforme eu precisei fazer testes. Depois, chama dentro do loop principal <code>self.func</code>, com os mesmos parâmetros, mas para criar as imagens seguintes. Esta organização com <code>preFunc</code> e <code>func</code> foi necessária para melhor visualizar os dados, uma vez que o algoritmo de desenho que comecei a usar para testes não dava uma resposta visual rápida. Logo, usei preFunc para desenhar uma imagem e func para modificá-la, como por exemplo, movendo suas linhas para cima.</p>
<p>Como precisamos trocar as imagens o mais rápido possível (30 frames por segundo = ~33 ms entre cada imagem), <code>run</code> deve executar seu loop o mais rápido possível.</p>
<p>O passo seguinte, que independe de como a imagem foi criada, é transformar a matriz de pontos numa imagem. Esta transformação é realizada por <code>image = Image.fromarray(data)</code>. A partir deste ponto, temos uma imagem, porém esta está no formato da PILLOW (PIL), biblioteca de imagens que usamos para fazer esta gestão. Para converter nossa imagem para o tkinter, também usando a PILLOW, chamamos: <code>converted_image = ImageTk.PhotoImage(image)</code>. <code>converted_image</code> está pronta para ir para a fila e ser desenhada na tela. Já podemos passar para o desenho da imagem seguinte.</p>
<p>Uma curiosidade é que foi justamente a ImageTk.PhotoImage que complicou o processo de finalização dos threads, é esta classe que precisa do loop de eventos do tkinter rodando para funcionar e fez com que uma coordenação de finalização fosse elaborada com o <code>queueStop</code>.</p>
<p>O loop de <code>run</code> fica rodando até <code>self.running</code> ser <code>False</code>. E é exatamente isto que o método <code>stop</code> faz.</p>
<p>Como o thread de desenho é independente do thread principal do programa, onde roda a tkinter, o <code>stop</code> pode ocorrer em momentos diferentes do loop. É por isso que não podemos desativar a tkinter até que o loop seja finalizado e chegue novamente no <code>while</code> que verifica <code>self.running</code>.</p>
<p>Ao sair do loop, uma mensagem a <code>queueStop</code> é postada. Esta mensagem serve de sinal para que o loop principal continue sua finalização e posteriormente feche a janela.</p>
<p>Você deve ter reparado várias chamadas comentadas a <code>TimeIt</code>. Esta classe foi criada apenas para medir o tempo de execução de algumas funções, pois percebi que estava muito lento.</p>
<div class="highlight"><pre><code class="language-python"><span>class</span> <span>TimeIt</span>:
<span>"""Classe para medir o tempo de execução de alguns blocos.
</span><span> Deve ser usada como gerenciados de contexto, com blocks with"""</span>
<span>def</span> __init__(self, name, silent<span>=</span>False):
self<span>.</span>name <span>=</span> name
self<span>.</span>start <span>=</span> <span>0</span>
self<span>.</span>end <span>=</span> <span>0</span>
self<span>.</span>silent <span>=</span> silent
<span>def</span> __enter__(self):
self<span>.</span>start <span>=</span> datetime<span>.</span>now()
<span>def</span> __exit__(self, <span>*</span>args, <span>**</span>kwargs):
self<span>.</span>end <span>=</span> datetime<span>.</span>now()
<span>if</span> <span>not</span> self<span>.</span>silent:
segundos <span>=</span> self<span>.</span>elapsed()<span>.</span>total_seconds()
<span>if</span> segundos <span>==</span> <span>0</span>:
<span>return</span>
fps <span>=</span> <span>1.0</span> <span>/</span> segundos
<span>print</span>(f<span>"Elapsed {self.name}: {self.elapsed()} Frames: {fps}"</span>)
<span>def</span> <span>elapsed</span>(self):
<span>return</span> self<span>.</span>end <span>-</span> self<span>.</span>start
</code></pre></div><p>Antes de tudo otimizar era preciso descobrir a origem da lentidão. No caso do loop, era sempre a chamada a <code>self.func</code> que dominava o tempo de execução. Você pode remover os comentários e identar a linha seguinte para ter os resultados na tela. A operação de <code>Image.fromarray</code> e <code>ImageTk.PhotoImage</code> executam muito rápido, na casa de 1 ms em meu computador. Já a função de desenho estava demorando até 3s ou 3000 ms no início. Lembrando que precisamos desenhar em no máximo 33 ms para termos 30 frames por segundo.</p>
<p>Vejamos uma função simples de desenho:</p>
<div class="highlight"><pre><code class="language-python"><span>def</span> <span>draw</span>(data, c, largura, altura):
<span>for</span> y <span>in</span> numpy<span>.</span>arange(<span>0</span>, altura):
<span>for</span> x <span>in</span> numpy<span>.</span>arange(<span>0</span>, largura):
data[y, x] <span>=</span> [<span>0</span>, <span>0</span>, y <span>//</span> (c <span>+</span> <span>1</span>)]
</code></pre></div><p>Esta função simplesmente desenha uma série de listras na tela, mudando a componente azul de cada ponto com a divisão da linha corrente pelo contador de frames (<code>c</code>). A ideia era apenas de percorrer os pontos da imagem e poder visualizar na tela o mais rápido possível.</p>
<p>Esta função tem uma performance horrível:</p>
<pre><code>Elapsed Loop: 0:00:01.427400 Frames: 0.7005744710662744
Elapsed Loop: 0:00:01.316119 Frames: 0.7598097132554122
Elapsed Loop: 0:00:01.308270 Frames: 0.764368211454822
Elapsed Loop: 0:00:01.341486 Frames: 0.7454419949220491
Elapsed Loop: 0:00:01.359058 Frames: 0.7358037699641957
</code></pre><p>Menos de 1 frame por segundo, já que estamos passando mais de 1s para gerar uma única imagem.</p>
<p><img alt="Tela Azul" src="https://blog.nilo.pro.br/posts/images/efeitos/tela_azul_1.png" title="Tela azul" /></p>
<p>Como tempo, a imagem fica cada vez mais escura, em função dos valores de <code>c</code> que aumentam a cada frame. Mas com esta velocidade, ficou muito lento e você mal percebe que há alguma mudança na tela em si.</p>
<p>Mesmo usando NumPy, o tempo de execução do loop de desenho era muito alto. Decidi então usar outra biblioteca, chamada <a href="http://numba.pydata.org/">Numba</a>. Numba é um JIT (Just in Time compiler) para Python. Com ela, você pode anotar suas funções e elas são compiladas logo na primeira chamada. Ao chamar outra vez, a função original é substituída pela compilada, otimizada e rodando com performances de linguagens nativas (desde que a interação com o interpretador seja limitada).
Vejamos o que precisamos mudar para usar Numba:</p>
<div class="highlight"><pre><code class="language-python"><span>@jit</span>(nopython<span>=</span>True, parallel<span>=</span>True, fastmath<span>=</span>True, nogil<span>=</span>True)
<span>def</span> <span>drawNumba</span>(data, c, largura, altura):
<span>for</span> y <span>in</span> numpy<span>.</span>arange(<span>0</span>, altura):
<span>for</span> x <span>in</span> numpy<span>.</span>arange(<span>0</span>, largura):
data[y, x] <span>=</span> [<span>0</span>, <span>0</span>, y <span>//</span> (c <span>+</span> <span>1</span>)]
</code></pre></div><p>O código é o mesmo, simplesmente adicionamos o decorador <code>@jit</code> da Numba para marcar que queremos que esta função seja otimizada. Nada mais no código foi mudado, salvo o import da Numba em si. Para o resto do programa, a função se comporta da mesma forma que antes.</p>
<p>Vejamos o resultado com Numba:</p>
<pre><code>Elapsed PreLoop: 0:00:01.276058 Frames: 0.7836634384957424
Elapsed Loop: 0:00:00.210905 Frames: 4.74147127853773
Elapsed Loop: 0:00:00.205027 Frames: 4.877406390377853
Elapsed Loop: 0:00:00.219663 Frames: 4.552428037493797
Elapsed Loop: 0:00:00.220664 Frames: 4.5317768190552155
Elapsed Loop: 0:00:00.209917 Frames: 4.763787592238837
Elapsed Loop: 0:00:00.222617 Frames: 4.492019926600395
</code></pre><p>Eu adicionei um contexto TimeIt para medir o tempo de execução da primeira chamada a função, no caso, <code>preFunc</code>. Veja que a função executou praticamente com a mesma lentidão da versão sem aceleração. Porém, observe que a partir da segunda chamada, o tempo de execução foi reduzido de 1.27s para 0.21s, elevando nossos frames por segundo a 4.7 (o número de frames na tela pode ser menor ou um pouco diferente devido a comunicação com a tkinter). A versão acelerada com Numba, roda em apenas 16% do tempo, ou seja, é quase 8 vezes mais rápida. Tudo isso com a instalação via pip e duas linhas no código. Mas 4 frames por segundo ainda é muito lento e longe dos 30 desejados. Lembrando que até agora nem comecei a fazer o efeito de chamas.</p>
<p>Outra alternativa é usar um módulo compilado, criado com <a href="https://cython.org/">Cython</a>. Cython (diferente de CPython), é um compilador que traduz um programa parecido com Python em um módulo C que o Python pode chamar.</p>
<p>Para usar Cython, precisamos fazer algumas mudanças mais importantes. Primeiro instalar o Cython e ter certeza que um compilador C/C++ está instalado na máquina. No Windows com Python 3.8, usei o Visual Studio 2019 sem problemas.</p>
<p>Um programa em Cython é escrito num arquivo com a extensão .pyx. Convertendo a função de desenho para Cython, temos:</p>
<div class="highlight"><pre><code class="language-cython"><span>import</span> numpy <span>as</span> np
<span>cimport</span> numpy <span>as</span> np
<span>cimport</span> cython
<span>from</span> libc.math <span>cimport</span> abs
<span>from</span> libc.stdlib <span>cimport</span> rand
<span>ctypedef</span> np<span>.</span>uint8_t DTYPE_t
<span>ctypedef</span> np<span>.</span>uint32_t DTYPE32_t
<span>@cython</span><span>.</span>boundscheck(False)
<span>@cython</span><span>.</span>wraparound(False)
<span>@cython</span><span>.</span>nonecheck(False)
<span>@cython</span><span>.</span>cdivision(True)
<span>def</span> <span>draw2</span>(np<span>.</span>ndarray[DTYPE_t, ndim<span>=</span><span>3</span>] data, int c, int max_x, int max_y):
<span>cdef</span> <span>int</span> <span>x</span>, <span>y</span>
<span>cdef</span> <span>int</span> <span>ic</span> <span>=</span> c
<span>cdef</span> <span>np</span>.<span>ndarray</span>[<span>DTYPE_t</span>, <span>ndim</span><span>=</span><span>3</span>] h <span>=</span> data
<span>cdef</span> <span>int</span> <span>cmax_y</span> <span>=</span> max_y, cmax_x <span>=</span> max_x
<span>for</span> y <span>in</span> range(cmax_y):
<span>for</span> x <span>in</span> range(cmax_x):
h[y, x, <span>0</span>] <span>=</span> <span>0</span>
h[y, x, <span>1</span>] <span>=</span> <span>0</span>
h[y, x, <span>2</span>] <span>=</span> y <span>/</span> (ic <span>+</span> <span>1</span>)
</code></pre></div><p>Muito parecido com Python e com C.</p>
<p>O Cython pede também a configuração de um setup.py para compilar o módulo.</p>
<div class="highlight"><pre><code class="language-python"><span>from</span> setuptools <span>import</span> setup
<span>from</span> Cython.Build <span>import</span> cythonize
<span>import</span> numpy
setup(
name<span>=</span><span>'Gerador de Telas'</span>,
ext_modules<span>=</span>cythonize(<span>"compute.pyx"</span>, annotate<span>=</span>True, language_level<span>=</span><span>3</span>),
include_dirs<span>=</span>[numpy<span>.</span>get_include()],
zip_safe<span>=</span>False,
)
</code></pre></div><p>E precisa ser compilado com:</p>
<pre><code>python setup.py build_ext --inplace
</code></pre><p>Mas os resultados são muito bons:</p>
<pre><code>Elapsed Loop: 0:00:00.023445 Frames: 42.65301770100235
Elapsed Loop: 0:00:00.022442 Frames: 44.55930843953302
Elapsed Loop: 0:00:00.023414 Frames: 42.70949004868882
Elapsed Loop: 0:00:00.024410 Frames: 40.96681687832855
Elapsed Loop: 0:00:00.023431 Frames: 42.67850283812044
Elapsed Loop: 0:00:00.022455 Frames: 44.533511467379206
Elapsed Loop: 0:00:00.023436 Frames: 42.66939750810719
</code></pre><p>Agora passamos de 4 para 40 frames por segundo e geramos uma nova imagem em apenas 23 ms!</p>
<p>Na realidade, ficou tão rápido que a imagem fica preta muito rápido. Para facilitar a visualização, uma outra função, chamada <code>drawUp</code> for criada. Para não ficar escurecendo a imagem, resolvi copiar as linhas de forma a rolar as linhas na tela, desta forma, o programa pode rodar por mais tempo, sem que a tela fique negra.</p>
<pre><code>@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
@cython.cdivision(True)
def drawUp(np.ndarray[DTYPE_t, ndim=3] data, int c, int max_x, int max_y):
cdef int x, y
cdef int ic = c
cdef np.ndarray[DTYPE_t, ndim=3] h = data
cdef int cmax_y = max_y, cmax_x = max_x
# Copy top to bottom
for x in range(0, cmax_x):
h[cmax_y - 2, x, 0] = h[0, x, 0]
h[cmax_y - 2, x, 1] = h[0, x, 1]
h[cmax_y - 2, x, 2] = h[0, x, 2]
for y in range(1, cmax_y - 1):
for x in range(0, cmax_x):
h[y - 1, x, 0] = h[y, x, 0]
h[y - 1, x, 1] = h[y, x, 1]
h[y - 1, x, 2] = h[y, x, 2]
</code></pre><p>Foi está mudança que levou a separação entre <code>preFunc</code> e <code>func</code>.
No preFunc, executada por <code>draw2</code>, uma imagem como a gerada em Python puro é criada. Na função <code>drawUp</code>, ela simplesmente rola as linhas da imagem, copiando a linha do topo para baixo e movendo as outras linhas para cima.</p>
<p>Neste ponto, tanto o problema de performance quanto de finalização da janela foram resolvidos. Falta apenas converter o algoritmo para gerar as flamas.</p>
<p>O primeiro passo é gerar uma paleta de cores compatíveis, uma vez que o algoritmo usa 256 cores para indicar a intensidade do fogo.</p>
<p>Convertendo para Python, temos algo como:</p>
<div class="highlight"><pre><code class="language-python"><span>def</span> <span>build_fire_palette</span>():
palette <span>=</span> numpy<span>.</span>zeros((<span>256</span>, <span>3</span>), dtype<span>=</span>numpy<span>.</span>uint8)
<span>for</span> x <span>in</span> range(<span>256</span>):
h <span>=</span> x <span>//</span> <span>3</span>
saturation <span>=</span> <span>100</span>
b <span>=</span> min(<span>256</span>, x <span>*</span> <span>2</span>) <span>/</span> <span>256.0</span> <span>*</span> <span>100.0</span>
css <span>=</span> f<span>"hsl({h},{saturation}%,{b}%)"</span>
palette[x] <span>=</span> ImageColor<span>.</span>getrgb(css)
<span>return</span> palette
</code></pre></div><p>A paleta é simplesmente uma tabela de cores que vamos usar para transformar um valor entre 0 e 255 (byte) em uma cor RGB (vermelho, verde e azul com 3 bytes).</p>
<p>Um problema aperece com o desenhador, pois a classe <code>Desenha</code> não suporta paletas de cores. Vamos precisar de outro desenhador:</p>
<div class="highlight"><pre><code class="language-python"><span>class</span> <span>DesenhaComPalette</span>(Desenha):
<span>def</span> <span>run</span>(self):
<span>try</span>:
palette <span>=</span> build_fire_palette()
data <span>=</span> numpy<span>.</span>zeros((ALTURA, LARGURA), dtype<span>=</span>numpy<span>.</span>uint8)
fogo <span>=</span> numpy<span>.</span>zeros((ALTURA, LARGURA), dtype<span>=</span>numpy<span>.</span>uint32)
c <span>=</span> <span>0</span>
<span>while</span> self<span>.</span>running:
<span>with</span> TimeIt(<span>"Loop"</span>) <span>as</span> t:
<span># with TimeIt("ForLoop") as t:</span>
self<span>.</span>func(data, c, LARGURA, ALTURA, fogo)
<span># with TimeIt("FROM ARRAY") as t1:</span>
image <span>=</span> Image<span>.</span>fromarray(data, mode<span>=</span><span>"P"</span>)
image<span>.</span>putpalette(palette)
<span># with TimeIt("Convert") as t2:</span>
converted_image <span>=</span> ImageTk<span>.</span>PhotoImage(image)
<span># with TimeIt("Queue") as t3:</span>
self<span>.</span>queue<span>.</span>put((c, converted_image))
c <span>+=</span> <span>1</span>
<span>finally</span>:
self<span>.</span>running <span>=</span> False
self<span>.</span>queueStop<span>.</span>put((<span>0</span>, <span>'FEITO'</span>))
</code></pre></div><p>A diferença é que criamos a imagem de forma diferente, pois temos que passar os pontos (com cores 0 a 255) e a paleta (com a tradução de cada cor). Criamos também o fogo, mas como uma matriz de inteiros e não como uma matriz de bytes. Isto muda o tamanho da matriz em memória, mas é necessária pro algoritmo das flamas que guarda a informação do fogo entre uma tela e outra. Em <code>data</code>, vamos guardar os pontos em 256 cores.</p>
<p>O algoritmo convertido em Python fica assim:</p>
<pre><code>def desenhaPythonFlamas(data, c, largura, altura, fogo):
for x in range(LARGURA):
fogo[ALTURA - 1, x] = int(min(random.random() * 2048, 2048))
for y in range(1, ALTURA - 2):
for x in range(0, LARGURA):
v = int((fogo[(y + 1) % ALTURA, x] +
fogo[(y + 1) % ALTURA, (x - 1) % LARGURA] +
fogo[(y + 1) % ALTURA, (x + 1) % LARGURA] +
fogo[(y + 2) % ALTURA, x]) * 32) / 129
fogo[y, x] = v
for y in range(altura):
for x in range(largura):
data[y, x] = fogo[y, x] % 256
</code></pre><p>Que fica ultra lento, como esperado:</p>
<pre><code>Elapsed Loop: 0:00:06.345203 Frames: 0.15759937073723254
Elapsed Loop: 0:00:06.327644 Frames: 0.15803670370836284
Elapsed Loop: 0:00:06.362772 Frames: 0.15716420453223848
Elapsed Loop: 0:00:06.387171 Frames: 0.15656383710409505
Elapsed Loop: 0:00:06.590262 Frames: 0.15173903556489862
</code></pre><p>São enormes 6s para gerar uma só tela com as flamas! Passemos a versão otimizada com Numba, simplesmente adicionado o decorador, como fizemos anteriormente.</p>
<pre><code>Elapsed Loop: 0:00:00.022445 Frames: 44.55335263978615
Elapsed Loop: 0:00:00.024425 Frames: 40.941658137154555
Elapsed Loop: 0:00:00.024421 Frames: 40.94836411285369
Elapsed Loop: 0:00:00.024396 Frames: 40.99032628299721
</code></pre><p>Muito melhor! Atingimos mais de 30 frames como esperado, voltando a casa dos 23 ms para gerar um frame. Lembrando que todas estas performances são para imagens de 1024 x 1024 pontos. Se você tem um computador mais lento, pode diminuir o tamanho da tela.</p>
<p>E como ficaria em Cython? Pagando para ver:</p>
<pre><code>@cython.boundscheck(False)
@cython.wraparound(False)
@cython.nonecheck(False)
@cython.cdivision(True)
def desenhaflamas(np.ndarray[DTYPE_t, ndim=2] data,
int c, int max_x, int max_y,
np.ndarray[DTYPE32_t, ndim=2] fogo):
cdef int x, y
cdef int ic = c
cdef np.ndarray[DTYPE_t, ndim=2] d = data
cdef np.ndarray[DTYPE32_t, ndim=2] f = fogo
cdef int cmax_y = max_y, cmax_x = max_x
for x in range(cmax_x):
f[cmax_y - 1, x] = abs(32768 + rand()) % 2048
for y in range(1, cmax_y - 2):
for x in range(0, cmax_x):
f[y, x] = ((f[(y + 1) % cmax_y, x] +
f[(y + 1) % cmax_y, (x - 1) % cmax_x] +
f[(y + 1)% cmax_y, (x + 1) % cmax_x] +
f[(y + 2)% cmax_y, x]) * 32) / 129
for y in range(max_y):
for x in range(max_x):
d[y, x] = f[y, x] % 256
</code></pre><p>Que tem como performance:</p>
<pre><code>Elapsed Loop: 0:00:00.026373 Frames: 37.917567208887874
Elapsed Loop: 0:00:00.026379 Frames: 37.90894271958755
Elapsed Loop: 0:00:00.028309 Frames: 35.324455120279765
Elapsed Loop: 0:00:00.030253 Frames: 33.05457310018841
Elapsed Loop: 0:00:00.028315 Frames: 35.316969803990816
Elapsed Loop: 0:00:00.026374 Frames: 37.91612952149844
Elapsed Loop: 0:00:00.026347 Frames: 37.95498538733063
Elapsed Loop: 0:00:00.026374 Frames: 37.91612952149844
</code></pre><p>Ficou um pouco pior que com Numba. Eu acredito que seja algum detalhe no código Cython. Mas já deu para ver que o efeito está rodando:</p>
<p><img alt="Flamas" src="https://blog.nilo.pro.br/posts/images/efeitos/flamas_em_python.png" title="Flamas em Python" /></p>
<p>Como rodar tudo isso em um só programa? Precisamos de uma seção de configuração:</p>
<div class="highlight"><pre><code class="language-python"><span>if</span> len(sys<span>.</span>argv) <span><</span> <span>5</span>:
<span>print</span>(<span>"Uso: python desenha.py <algoritmo> <acelerador> <largura> <altura>"</span>)
<span>print</span>(<span>"Algoritmo: desenho, flamas"</span>)
<span>print</span>(<span>"Acelerador: cython, python, numba"</span>)
ALGORITMO <span>=</span> sys<span>.</span>argv[<span>1</span>]<span>.</span>lower()
ACELERADOR <span>=</span> sys<span>.</span>argv[<span>2</span>]<span>.</span>lower()
LARGURA <span>=</span> int(sys<span>.</span>argv[<span>3</span>])
ALTURA <span>=</span> int(sys<span>.</span>argv[<span>4</span>])
<span>print</span>(f<span>"ALGORITMO: {ALGORITMO}"</span>)
<span>print</span>(f<span>"ACELERADOR: {ACELERADOR}"</span>)
<span>print</span>(f<span>"LARGURA: {LARGURA} ALTURA: {ALTURA}"</span>)
CONFIGURACAO <span>=</span> {
<span>"flamas"</span>: {<span>"desenhador"</span>: DesenhaComPalette,
<span>"otimizacao"</span>: {<span>"python"</span>: (desenhaPythonFlamas<span>.</span>py_func, None),
<span>"cython"</span>: (desenhaflamas, None),
<span>"numba"</span>: (desenhaPythonFlamas, None)
}},
<span>"desenho"</span>: {<span>"desenhador"</span>: Desenha,
<span>"otimizacao"</span>: {<span>"python"</span>: (drawNumba<span>.</span>py_func, None),
<span>"cython"</span>: (drawUp, draw2),
<span>"numba"</span>: (drawNumba, None)
}}
}
<span>if</span> ALGORITMO <span>not</span> <span>in</span> CONFIGURACAO:
<span>print</span>(f<span>"Algoritmo {ALGORITMO} inválido"</span>, file<span>=</span>sys<span>.</span>stderr)
sys<span>.</span>exit(<span>1</span>)
<span>if</span> ACELERADOR <span>not</span> <span>in</span> CONFIGURACAO[ALGORITMO][<span>"otimizacao"</span>]:
<span>print</span>(f<span>"Acelerador {ACELERADOR} inválido"</span>, file<span>=</span>sys<span>.</span>stderr)
sys<span>.</span>exit(<span>2</span>)
<span>if</span> ALTURA <span><</span> MIN_V <span>or</span> LARGURA <span><</span> MIN_V <span>or</span> ALTURA <span>></span> MAX_V <span>or</span> LARGURA <span>></span> MAX_V:
<span>print</span>(f<span>"Altura e largura devem ser valores entre {MIN_V} e {MAX_V}."</span>)
sys<span>.</span>exit(<span>3</span>)
desenhador <span>=</span> CONFIGURACAO[ALGORITMO][<span>"desenhador"</span>]
func <span>=</span> CONFIGURACAO[ALGORITMO][<span>"otimizacao"</span>][ACELERADOR][<span>0</span>]
prefunc <span>=</span> CONFIGURACAO[ALGORITMO][<span>"otimizacao"</span>][ACELERADOR][<span>1</span>]
app <span>=</span> App(desenhador<span>=</span>desenhador, func<span>=</span>func, preFunc<span>=</span>prefunc)
app<span>.</span>mainloop()
</code></pre></div><p>Ufa, finalmente rodando! Espero que tenha gostado do artigo e que tenha ficado curioso sobre performance em Python, Cython e Numba. O código completo esta publicado no GitHub:
<a href="https://github.com/lskbr/flamas_em_python">https://github.com/lskbr/flamas_em_python</a></p>
<p>E se usarmos este código para simular o <a href="https://pt.wikipedia.org/wiki/Jogo_da_vida">Jogo da Vida</a>? Fica para outro artigo.</p>Nilo Ney Coutinho Menezeshttps://blog.nilo.pro.br/tags/python/Pegando dados macroeconômicos com Quandl em Pythonhttps://paulohrpinheiro.xyz/texts/python/2020-05-20-pegando-dados-macroeconomicos-com-quandl-em-python.html2020-05-20T00:00:00+00:00Como obter dados históricos sobre indicadores de macroeconomia com a biblioteca Quandl usando PythonBlog do PauloHRPinheirohttps://paulohrpinheiro.xyzPutting it all togethertag:blog.luizirber.org,2020-05-11:/2020/05/11/sbt-zip/2020-05-11T15:00:00+00:00<p>sourmash <a href="https://twitter.com/ctitusbrown/status/1257418140729868291">3.3</a> was released last week,
and it is the first version supporting <a href="http://ivory.idyll.org/blog/2020-sourmash-databases-as-zip-files.html">zipped databases</a>.
Here is my personal account of how that came to be =]</p>
<h2>What is a sourmash database?</h2>
<p>A sourmash database contains signatures (typically Scaled MinHash sketches built from genomic datasets) and
an index for allowing efficient similarity and containment queries over these signatures.
The two types of index are SBT,
a hierarchical index that uses less memory by keeping data on disk,
and LCA,
an inverted index that uses more memory but is potentially faster.
Indices are described as JSON files,
with LCA storing all the data in one JSON file and SBT opting for saving a description of the index structure in JSON,
and all the data into a hidden directory with many files.</p>
<p>We distribute some <a href="https://sourmash.readthedocs.io/en/v3.3.0/databases.html">prepared databases</a> (with SBT indices) for Genbank and RefSeq as compressed TAR files.
The compressed file is ~8GB,
but after decompressing it turns into almost 200k files in a hidden directory,
using about 40 GB of disk space.</p>
<h2>Can we avoid generating so many hidden files?</h2>
<p>The initial issue in this saga is <a href="https://github.com/dib-lab/sourmash/issues/490">dib-lab/sourmash#490</a>,
and the idea was to take the existing support for multiple data storages
(hidden dir,
TAR files,
IPFS and Redis) and save the index description in the storage,
allowing loading everything from the storage.
Since we already had the databases as TAR files,
the first test tried to use them but it didn't take long to see it was a doomed approach:
TAR files are terrible from random access
(or at least the <code>tarfile</code> module in Python is).</p>
<p>Zip files showed up as a better alternative,
and it helps that Python has the <code>zipfile</code> module already available in the
standard library.
Initial tests were promising,
and led to <a href="https://github.com/dib-lab/sourmash/pull/648">dib-lab/sourmash#648</a>.
The main issue was performance:
compressing and decompressing was slow,
but there was also another limitation...</p>
<h2>Loading Nodegraphs from a memory buffer</h2>
<p>Another challenge was efficiently loading the data from a storage.
The two core methods in a storage are <code>save(location, content)</code>,
where <code>content</code> is a bytes buffer,
and <code>load(location)</code>,
which returns a bytes buffer that was previously saved.
This didn't interact well with the <code>khmer</code> <code>Nodegraph</code>s (the Bloom Filter we use for SBTs),
since <code>khmer</code> only loads data from files,
not from memory buffers.
We ended up doing a temporary file dance,
which made things slower for the default storage (hidden dir),
where it could have been optimized to work directly with files,
and involved interacting with the filesystem for the other storages
(IPFS and Redis could be pulling data directly from the network,
for example).</p>
<p>This one could be fixed in <code>khmer</code> by exposing C++ stream methods,
and I did a <a href="https://github.com/luizirber/2018-cython-streams">small PoC</a> to test the idea.
While doable,
this is something that was happening while the sourmash conversion to Rust was underway,
and depending on <code>khmer</code> was a problem for my Webassembly aspirations...
so,
having the Nodegraph <a href="https://github.com/luizirber/sourmash-rust/pull/15">implemented in Rust</a> seemed like a better direction,
That has actually been quietly living in the sourmash codebase for quite some time,
but it was never exposed to the Python (and it was also lacking more extensive
tests).</p>
<p>After the release of sourmash 3 and the replacement of the C++ for the Rust implementation,
all the pieces for exposing the Nodegraph where in place,
so <a href="https://github.com/dib-lab/sourmash/pull/799">dib-lab/sourmash#799</a> was the next step.
It wasn't a priority at first because other optimizations
(that were released in 3.1 and 3.2)
were more important,
but then it was time to check how this would perform.
And...</p>
<h2>Your Rust code is not so fast, huh?</h2>
<p>Turns out that my Nodegraph loading code was way slower than <code>khmer</code>.
The Nodegraph binary format <a href="https://khmer.readthedocs.io/en/latest/dev/binary-file-formats.html#nodegrap://khmer.readthedocs.io/en/latest/dev/binary-file-formats.html#nodegraph">is well documented</a>,
and doing an initial implementation wasn't so hard by using the <code>byteorder</code> crate
to read binary data with the right endianess,
and then setting the appropriate bits in the internal <code>fixedbitset</code> in memory.
But the khmer code doesn't parse bit by bit:
it <a href="https://github.com/dib-lab/khmer/blob/fe0ce116456b296c522ba24294a0cabce3b2648b/src/oxli/storage.cc#L233">reads</a> a long <code>char</code> buffer directly,
and that is many orders of magnitude faster than setting bit by bit.</p>
<p>And there was no way to replicate this behavior directly with <code>fixedbitset</code>.
At this point I could either bit-indexing into a large buffer
and lose all the useful methods that <code>fixedbitset</code> provides,
or try to find a way to support loading the data directly into <code>fixedbitset</code> and
open a PR.</p>
<p><a href="https://github.com/petgraph/fixedbitset/pull/42">I chose the PR</a> (and even got #42! =]).</p>
<p>It was more straightforward than I expected,
but it did expose the internal representation of <code>fixedbitset</code>,
so I was a bit nervous it wasn't going to be merged.
But <a href="https://github.com/bluss">bluss</a> was super nice,
and his suggestions made the PR way better!
This <a href="https://github.com/dib-lab/sourmash/blob/9a695fb03b99c060bb8d1384ab78bb3797c5eb65/src/core/src/sketch/nodegraph.rs#L235L261">simplified</a> the final <code>Nodegraph</code> code,
and actually was more correct
(because I was messing a few corner cases when doing the bit-by-bit parsing before).
Win-win!</p>
<h2>Nodegraphs are kind of large, can we compress them?</h2>
<p>Being able to save and load <code>Nodegraph</code>s in Rust allowed using memory buffers,
but also opened the way to support other operations not supported in khmer <code>Nodegraph</code>s.
One example is loading/saving compressed files,
which is supported for <code>Countgraph</code>
(another khmer data structure,
based on Count-Min Sketch)
but not in <code>Nodegraph</code>.</p>
<p>If only there was an easy way to support working with compressed files...</p>
<p>Oh wait, there is! <a href="https://github.com/luizirber/niffler">niffler</a> is a crate that I made with <a href="https://twitter.com/pierre_marijon">Pierre Marijon</a> based
on some functionality I saw in one of his projects,
and we iterated a bit on the API and documented everything to make it more
useful for a larger audience.
<code>niffler</code> tries to be as transparent as possible,
with very little boilerplate when using it but with useful features nonetheless
(like auto detection of the compression format).
If you want more about the motivation and how it happened,
check this <a href="https://twitter.com/luizirber/status/1253445504622424064">Twitter thread</a>.</p>
<p>The cool thing is that adding compressed files support in <code>sourmash</code> was mostly
<a href="https://github.com/dib-lab/sourmash/pull/799/files#diff-313a7ff0fdb14f408a64b3f010f46f65R220">one-line changes</a> for loading
(and <a href="https://github.com/dib-lab/sourmash/pull/648/files#diff-d80ae1dd777d07300d7b6066b3318397L249-R273">a bit more</a> for saving,
but mostly because converting compression levels could use some refactoring).</p>
<h2>Putting it all together: zipped SBT indices</h2>
<p>With all these other pieces in places,
it's time to go back to <a href="https://github.com/dib-lab/sourmash/pull/648">dib-lab/sourmash#648</a>.
Compressing and decompressing with the Python <code>zipfile</code> module is slow,
but Zip files can also be used just for storage,
handing back the data without extracting it.
And since we have compression/decompression implemented in Rust with <code>niffler</code>,
that's what the zipped sourmash databases are:
data is loaded and saved into the Zip file without using the Python module
compression/decompression,
and all the work is done before (or after) in the Rust side.</p>
<p>This allows keeping the Zip file with similar sizes to the original TAR files we started with,
but with very low overhead for decompression.
For compression we opted for using Gzip level 1,
which doesn't compress perfectly but also doesn't take much longer to run:</p>
<table>
<thead>
<tr>
<th>Level</th>
<th>Size</th>
<th>Time</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>407 MB</td>
<td>16s</td>
</tr>
<tr>
<td>1</td>
<td>252 MB</td>
<td>21s</td>
</tr>
<tr>
<td>5</td>
<td>250 MB</td>
<td>39s</td>
</tr>
<tr>
<td>9</td>
<td>246 MB</td>
<td>1m48s</td>
</tr>
</tbody>
</table>
<p>In this table, <code>0</code> is without compression,
while <code>9</code> is the best compression.
The size difference from <code>1</code> to <code>9</code> is only 6 MB (~2% difference)
but runs 5x faster,
and it's only 30% slower than saving the uncompressed data.</p>
<p>The last challenge was updating an existing Zip file.
It's easy to support appending new data,
but if any of the already existing data in the file changes
(which happens when internal nodes change in the SBT,
after a new dataset is inserted) then there is no easy way to replace the data in the Zip file.
Worse,
the Python <code>zipfile</code> will add the new data while keeping the old one around,
leading to ginormous files over time<sup id="sf-sbt-zip-1-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sbt-zip-1" title="The zipfile module does throw a UserWarning pointing that duplicated files were inserted, which is useful during development but generally doesn't show during regular usage...">1</a></sup>
So, what to do?</p>
<p>I ended up opting for dealing with the complexity and <a href="https://github.com/dib-lab/sourmash/pull/648/files#diff-a99b088adcc872e1b408fbdcca20ebebR110-R248">complicating the ZipStorage</a> implementation a bit,
by keeping a buffer for new data.
If it's a new file or it already exists but there are no insertions
the buffer is ignored and all works as before.</p>
<p>If the file exists and new data is inserted,
then it is first stored in the buffer
(where it might also replace a previous entry with the same name).
In this case we also need to check the buffer when trying to load some data
(because it might exist only in the buffer,
and not in the original file).</p>
<p>Finally,
when the <code>ZipStorage</code> is closed it needs to verify if there are new items in the buffer.
If not,
it is safe just to close the original file.
If there are new items but they were not present in the original file,
then we can append the new data to the original file.
The final case is if there are new items that were also in the original file,
and in this case a new Zip file is created and all the content from buffer and
original file are copied to it,
prioritizing items from the buffer.
The original file is replaced by the new Zip file.</p>
<p>Turns out this worked quite well! And so the PR was merged =]</p>
<h2>The future</h2>
<p>Zipped databases open the possibility of distributing extra data that might be
useful for some kinds of analysis.
One thing we are already considering is adding <a href="https://github.com/dib-lab/sourmash/issues/969">taxonomy information</a>,
let's see what else shows up.</p>
<p>Having <code>Nodegraph</code> in Rust is also pretty exciting,
because now we can change the internal representation for something that uses
less memory (maybe using <a href="https://alexbowe.com/rrr/">RRR encoding</a>?),
but more importantly:
now they can also be used with Webassembly,
which opens many possibilities for running not only <a href="https://blog.luizirber.org/2018/08/27/sourmash-wasm/">signature computation</a> but
also <code>search</code> and <code>gather</code> in the browser,
since now we have all the pieces to build it.</p>
<h2>Comments?</h2>
<ul>
<li><a href="https://twitter.com/luizirber/status/1260031886744621059">Thread on Twitter</a></li>
</ul><hr /><h2>Footnotes</h2><ol><li id="sf-sbt-zip-1"><p>The <code>zipfile</code> module does throw a <code>UserWarning</code> pointing that duplicated files were inserted,
which is useful during development but generally doesn't show during regular usage... <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sbt-zip-1-back">↩</a></p></li></ol>luizirberhttps://blog.luizirber.org/LaKademy 2019https://blog.filipesaraiva.info/?p=21512020-05-05T18:29:16+00:00<p>Em novembro passado, colaboradores latinoamericanos do KDE desembarcaram em Salvador/Brasil para participarem de mais uma edição do LaKademy – o <em>Latin American Akademy</em>. Aquela foi a sétima edição do evento (ou oitava, se você contar o Akademy-BR como o primeiro LaKademy) e a segunda com Salvador como a cidade que hospedou o evento. Sem problemas para mim: na verdade, adoraria me mudar e viver ao menos alguns anos em Salvador, cidade que gosto baste. <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/13.0.1/72x72/1f642.png" /></p>
<div class="wp-caption aligncenter" id="attachment_2145"><img alt="" class=" wp-image-2145" height="525" src="http://blog.filipesaraiva.info/wp-content/uploads/2020/04/photo_2020-04-16_11-55-14-1024x768.jpg" width="700" /><p class="wp-caption-text" id="caption-attachment-2145">Foto em grupo do LaKademy 2019</p></div>
<p>Minhas principais tarefas durante o evento foram em 2 projetos: Cantor e Sprat, um “editor de rascunhos de artigos acadêmicos”. Além deles, ajudei também com tarefas de promoção como o site do LaKademy.</p>
<p>Nos trabalhos sobre o Cantor me foquei naqueles relacionados com organização. Por exemplo, pedi aos sysadmins que <a href="https://invent.kde.org/kde/cantor" rel="noopener noreferrer" target="_blank">migrassem o repositório para o Gitlab</a> do KDE e crei um site específico para o Cantor em <a href="https://cantor.kde.org/" rel="noopener noreferrer" target="_blank">cantor.kde.org</a> usando o novo <a href="https://invent.kde.org/websites/jekyll-kde-theme" rel="noopener noreferrer" target="_blank">template em Jekyll para projetos do KDE</a>.</p>
<p>O novo site é uma boa adição ao Cantor porque nós queremos comunicar melhor e mais diretamente com nossa comunidade de usuários. O site tem um <a href="https://cantor.kde.org/blog/" rel="noopener noreferrer" target="_blank">blog</a> próprio e uma seção de <a href="https://cantor.kde.org/changelogs/" rel="noopener noreferrer" target="_blank">changelog</a> para tornar mais fácil à comunidde seguir as notícias e principais mudanças no software.</p>
<p>A migração para o Gitlab nos permite utilizar o Gitlab CI como uma alternativa para integração contínua no Cantor. Eu orientei o trabalho do Rafael Gomes (que ainda <a href="https://invent.kde.org/kde/cantor/-/merge_requests/1" rel="noopener noreferrer" target="_blank">não teve merge</a>) para termos isso disponível pro projeto.</p>
<p>Além dos trabalhos no Cantor, desenvolvi algumas atividades relacionadas ao Sprat, um editor de rascunhos de artigos científicos em inglês. Este softwar usa katepart para implementar e metodologia de escrita de artigos científicos em inglês conhecida como PROMETHEUS, conforme descrita <a href="https://www.amazon.com/Writing-Scientific-Papers-English-Successfully/dp/8588533979" rel="noopener noreferrer" target="_blank">neste livro</a>, como uma tentativa de auxiliar estudantes e pesquisadores em geral na tarefa de escrever artigos científicos. Durante o LaKademy finalizei o port para Qt5 e, tomara, espero lançar o projeto este ano.</p>
<p>Nas atividades mais sociais, participei da famosíssima reunião de promo, que discute as ações futuras do KDE para a América Latina. Nossa principal decisão foi organizar e participar mais de eventos pequenos e distribuídos em várias cidades, marcando a presença do KDE em eventos consolidados como o FLISoL e o Software Freedom Day, e mais – mas agora, em tempos de COVID-19, isso não é mais viável. Outra decisão foi mover a organização do KDE Brasil do Phabricator para o Gitlab.</p>
<div class="wp-caption aligncenter" id="attachment_2143"><img alt="" class="wp-image-2143" height="428" src="http://blog.filipesaraiva.info/wp-content/uploads/2020/04/photo_2020-04-16_11-56-11-1024x768.jpg" width="571" /><p class="wp-caption-text" id="caption-attachment-2143">Contribuidores do KDE trabalhando pesado</p></div>
<p>Para além da parte técnica, este LaKademy foi uma oportunidade para encontrar velhos e novos amigos, beber algumas cervejas, saborear a maravilhosa cozinha bahiana, e se divertir entre um commit e outro. <img alt="🙂" class="wp-smiley" src="https://s.w.org/images/core/emoji/13.0.1/72x72/1f642.png" /></p>
<p>Gostaria de agradecer ao <a href="https://ev.kde.org/" rel="noreferrer noopener" target="_blank">KDE e.V.</a> por apoiar o LaKademy, e Caio e Icaro por terem organizado essa edição do evento. Não vejo a hora de participar do próximo LaKademy e que isso seja o mais rápido possível! <img alt="😉" class="wp-smiley" src="https://s.w.org/images/core/emoji/13.0.1/72x72/1f609.png" /></p>Filipe Saraivahttps://blog.filipesaraiva.infoAkademy 2019https://blog.filipesaraiva.info/?p=21492020-05-04T21:20:54+00:00<div class="wp-caption aligncenter" id="attachment_2132"><a href="http://blog.filipesaraiva.info/?attachment_id=2132" rel="attachment wp-att-2132"><img alt="" class="wp-image-2132" height="240" src="http://blog.filipesaraiva.info/wp-content/uploads/2020/02/akademy2019-groupphoto_1500-1024x372.jpg" width="661" /></a><p class="wp-caption-text" id="caption-attachment-2132">Foto em grupo do Akademy 2019</p></div>
<p>Em setembro de 2019 a cidade italiana de Milão sediou o principal encontro mundial dos colaboradores do KDE – o Akademy, onde membros de diferentes áreas como tradutores, desenvolvedores, artistas, pessoal de promo e mais se reúnem por alguns dias para pensar e construir o futuro dos projetos e comunidade(s) do KDE</p>
<p>Antes de chegar ao Akademy tomei um voo do Brasil para Portugal a fim de participar do <a href="https://epia2019.utad.pt/" rel="noopener noreferrer" target="_blank">EPIA</a>, uma conferência sobre inteligência artificial que aconteceu na pequena cidade de Vila Real, região do Porto. Após essa atividade acadêmica, voei do Porto à Milão e iniciei minha participação no Akademy 2019.</p>
<p>Infelizmente pousei no final da primeira manhã do evento, o que me fez perder apresentações interessantes sobre Qt 6 e os novos KDE’ Goals. Pela tarde pude algumas palestras sobre temas que também me chamam atenção, como Plasma para dispositivos móveis, MyCroft para a indústria automotiva, relatório do KDE e.V. e um <em>showcase</em> de estudantes do Google Summer of Code e do Season of KDE – muito legal ver projetos incríveis desenvolvidos pelos novatos.</p>
<p>No segundo dia me chamou atenção as palestras sobre o KPublicTransportation, LibreOffice para o Plasma, Get Hot New Stuffs – que imagino utilizarei em um futuro projeto – e a apresentação do <a href="https://caiojcarvalho.wordpress.com/2020/02/14/akademy-2019-late-report/" rel="noopener noreferrer" target="_blank">Caio sobre o kpmcore</a>.</p>
<p>Após a festa do evento (comentários sobre ela apenas pessoalmente), os dias seguintes foram preenchidos pelos BoFs, para mim a parte mais interessante do Akademy.</p>
<p>O workshop do Gitlab foi interessante porque pudemos discutir temas específicos sobre a migração do KDE para esta ferramenta. Estou adorando esse movimento e espero que todos os projetos do KDE façam essa migração o quanto antes. <a href="https://invent.kde.org/kde/cantor" rel="noopener noreferrer" target="_blank">Cantor já está por lá</a> há algum tempo.</p>
<p>No BoF do KDE websites, pude entender um pouco melhor sobre o novo tema do Jekyll utilizado pelos nossos sites. Em adição, aguardo que logo mais possamos aplicar internacionalização nessas páginas, tornando-as traduzíveis para quaisquer idiomas. Após participar e tomar informações nesse evento, <a href="https://cantor.kde.org/" rel="noopener noreferrer" target="_blank">criei um novo website pro Cantor</a> durante o LaKademy 2019.</p>
<p>O BoF do KDE Craft foi interessante para ver como compilar e distribuir nosso software na loja do Windows (pois é, vejam só o que estou escrevendo…). Espero trabalhar com esse tema durante o ano de forma a disponibilizar um pacote do Cantor naquela loja ((pois é)²).</p>
<p>Também participei do workshop sobre QML e Kirigami realizado pelo pessoal do projeto <a href="https://mauikit.org/" rel="noopener noreferrer" target="_blank">Maui</a>. Kirigami é algo que tenho mantido o olho para futuros projetos.</p>
<p>Finalmente, participei do BoF “All About the Apps Kick Off”. Pessoalmente, penso que esse é o futuro do KDE: uma comunidade internacional que produz software livre de alta qualidade e seguro para diferentes plataformas, do desktop ao mobile. De fato, isto é como o KDE está atualmente organizado e funcionando, mas nós não conseguimos comunicar isso muito bem para o público. Talvez, com as mudanças em nossa forma de lançamentos, somados aos websites para projetos específicos e distribuição em diferentes lojas de aplicativos, possamos mudar a maneira como o público vê a nossa comunidade.</p>
<p>O day trip do Akademy 2019 foi no lago Como, na cidade de Varenna. Uma viagem linda, passei o tempo todo imaginando que lá poderia ser um bom lugar para eu passar uma lua de mel :D. Espero voltar lá no futuro próximo, e passar alguns dias viajando entre cidades como ela.</p>
<div class="wp-caption aligncenter" id="attachment_2134"><img alt="" class="wp-image-2134" height="412" src="http://blog.filipesaraiva.info/wp-content/uploads/2020/02/filipe-italia-1024x767.jpg" width="550" /><p class="wp-caption-text" id="caption-attachment-2134">Eu em Varenna</p></div>
<p>Gostaria de agradecer a todo o time local, Riccardo e seus amigos, por organizarem essa edição incrível do Akademy. Milão é uma cidade muito bonita, com comida deliciosa (carbonara!), lugares históricos para visitar e descobrir mais sobre os italianos e a sofisticada capital da alta moda.</p>
<p>Finalmente, meus agradecimentos ao KDE e.V. por patrocinar minha participação no Akademy.</p>
<p><a href="https://files.kde.org/akademy/2019/" rel="noopener noreferrer" target="_blank">Neste link</a> estão disponíveis vídeos das apresentações e BoFs do Akademy 2019.</p>Filipe Saraivahttps://blog.filipesaraiva.infoConsultas do Telegram - Questões interessanteshttps://blog.nilo.pro.br/posts/2020-05-03-consultas-do-telegram-questoes-interessantes/2020-05-03T13:45:00+00:00<p>Semana passada tive a oportunidade de ver 3 questões interessantes a discutir nos grupos do Telegram, mas que a explicação seria grande demais para apresentar em um chat.</p>
<ol>
<li>
<p><a href="https://blog.nilo.pro.br/tags/python/index.xml#q1">Por que <code>True, True, True == (True, True, True)</code> retorna <code>True, True, False</code>?</a></p>
</li>
<li>
<p><a href="https://blog.nilo.pro.br/tags/python/index.xml#q2">Por que <code>-2 * 5 // 3 + 1</code> retorna <code>-3</code>?</a></p>
</li>
<li>
<p><a href="https://blog.nilo.pro.br/tags/python/index.xml#q3">Como adicionar tempos em Python?</a></p>
</li>
</ol>
<h1 id="q1">Por que <code>True, True, True == (True, True, True)</code> retorna <code>True, True, False</code>?</h1>
<p>Esta questão foi apresentada como sendo uma sintaxe bizarra do Python, mas na realidade é uma pegadinha visual. Repare no operador <code>==</code> (igual igual).</p>
<div class="highlight"><pre><code class="language-python">Python <span>3.8</span><span>.</span><span>2</span> (default, Apr <span>27</span> <span>2020</span>, <span>15</span>:<span>53</span>:<span>34</span>)
[GCC <span>9.3</span><span>.</span><span>0</span>] on linux
Type <span>"help"</span>, <span>"copyright"</span>, <span>"credits"</span> <span>or</span> <span>"license"</span> <span>for</span> more information<span>.</span>
<span>>>></span> True, True, True <span>==</span> (True, True, True)
(True, True, False)
</code></pre></div><p>Algumas pessoas ficaram em dúvida e perguntaram por que (True, True, False) em vez de (True, True, True).
Primeiramente, devemos lembrar que Python permite criarmos tuplas sem utilizar parênteses, apenas com vírgulas. E é exatamente esta sintaxe que causou certa confusão, pois olhando rápido você pode pensar que estamos comparando duas tuplas, o que não é o caso. Por exemplo:</p>
<div class="highlight"><pre><code class="language-python"><span>>>></span> (True, True, True) <span>==</span> (True, True, True)
True
</code></pre></div><p>A diferença são os parênteses. Sem os parênteses, estamos comparando apenas <code>True == (True, True, True)</code> e não a primeira tupla com a segunda. É uma questão de prioridade de operadores. Desta forma, para criar a tupla, o interpretador avalia primeiramente a comparação de <code>True</code> com <code>(True, True, True)</code>. Como o primeiro é do tipo <code>bool</code> e o segundo uma tupla, o resultado é <code>False</code>. Assim, a tupla gerada tem os primeiros <code>True, True</code>, seguidos do <code>False</code> que é o resultado da comparação.
Quando escrevemos entre parênteses, estamos comparando duas tuplas e o resultado é <code>True</code>.</p>
<h1 id="q2">Por que <code>-2 * 5 // 3 + 1</code> retorna <code>-3</code>?</h1>
<p>Esta aqui é mais complicada. Vários resultados foram apresentados e a prioridade do operador <code>//</code> foi questionada. Vejamos o que diz o Python:</p>
<div class="highlight"><pre><code class="language-python"><span>>>></span> <span>-</span><span>2</span> <span>*</span> <span>5</span> <span>//</span> <span>3</span> <span>+</span> <span>1</span>
<span>-</span><span>3</span>
</code></pre></div><p>Qual resultado você esperava? Algumas pessoas esperavam -4, outras -2. Como resulta em -3?</p>
<p>Primeiro, devemos rever a prioridade do operador <code>//</code> que é exatamente a mesma da divisão. No caso, a divisão e a multiplicação tem a mesma prioridade e devem ser avalidas da esquerda para a direita. Isto é especialmente importante, pois o <code>//</code> faz um arredondamento.</p>
<p>Agora vejamos a definição do <code>//</code> na <a href="https://docs.python.org/3/reference/expressions.html#binary-arithmetic-operations">documentação do Python</a>:</p>
<blockquote>
<p>Division of integers yields a float, while floor division of integers results in an integer; the result is that of mathematical division with the ‘floor’ function applied to the result…</p>
</blockquote>
<p>Que podemos traduzir como:
A divisão de inteiros resulta em um número de ponto flutuante (float), enquanto a divisão piso(floor) de inteiros resulta em um número inteiro (int); o resultado é o da divisão matemática com a aplicação da função piso(floor) aplicada ao resultado.</p>
<p>O ponto que não ficou claro é o comportamento de <code>floor</code> com números negativos.</p>
<pre><code>>>> 10 // 3
3
>>> -10 // 3
-4
>>> -10 / 3
-3.3333333333333335
</code></pre><p>Naturalmente, se espera que o resultado de <code>-10 // 3</code> fosse igual ao de <code>10 // 3</code>, porém com sinal diferente. Você pode consultar a definição destas duas funções na <a href="https://en.wikipedia.org/wiki/Floor_and_ceiling_functions">Wikipedia</a> e na <a href="https://docs.python.org/3/library/math.html#math.floor">documentação do Python</a>:</p>
<blockquote>
<p>math.floor(x)
Return the floor of x, the largest integer less than or equal to x. If x is not a float, delegates to <code>x.__floor__()</code>, which should return an Integral value</p>
</blockquote>
<p>Que pode ser traduzido como: retorna o valor do piso de x, o maior número inteiro menor or igual a x. Se x não é um número de ponto flutuante, delega a <code>x.__floor()__</code> que deve retornar um valor inteiro.</p>
<p>Para números negativos, a parte de retornar o menor inteiro engana facilmente. Pois o menor inteiro relativo a -3.33 é -4, lembrando que -3 > -4!</p>
<p>Assim a expressão é corretamente avaliada pelo interpretador:</p>
<pre><code>-2 * 5
-10
-10 // 3
-4
-4 + 1
-3
</code></pre><p>Olhando os fontes do <a href="https://github.com/python/cpython/blob/v3.8.2/Objects/longobject.c#L3751">Python (3.8.2)</a> no github, fica fácil de ver o ajuste (linha 3768):</p>
<div class="highlight"><pre><code class="language-c"><span>3751</span><span>/* Fast floor division for single-digit longs. */</span>
<span>3752</span><span>static</span> PyObject <span>*</span>
<span>3753</span><span>fast_floor_div</span>(PyLongObject <span>*</span>a, PyLongObject <span>*</span>b)
<span>3754</span>{
<span>3755</span> sdigit left <span>=</span> a<span>-></span>ob_digit[<span>0</span>];
<span>3756</span> sdigit right <span>=</span> b<span>-></span>ob_digit[<span>0</span>];
<span>3757</span> sdigit div;
<span>3758</span>
<span>3759</span> assert(Py_ABS(Py_SIZE(a)) <span>==</span> <span>1</span>);
<span>3760</span> assert(Py_ABS(Py_SIZE(b)) <span>==</span> <span>1</span>);
<span>3761</span>
<span>3762</span> <span>if</span> (Py_SIZE(a) <span>==</span> Py_SIZE(b)) {
<span>3763</span> <span>/* 'a' and 'b' have the same sign. */</span>
<span>3764</span> div <span>=</span> left <span>/</span> right;
<span>3765</span> }
<span>3766</span> <span>else</span> {
<span>3767</span> <span>/* Either 'a' or 'b' is negative. */</span>
<span><span>3768</span> div <span>=</span> <span>-</span><span>1</span> <span>-</span> (left <span>-</span> <span>1</span>) <span>/</span> right;
</span><span>3769</span> }
<span>3770</span>
<span>3771</span> <span>return</span> PyLong_FromLong(div);
<span>3772</span>}
</code></pre></div><p>Nota: recordo de chamar a função piso de solo ou de mínima.</p>
<h1 id="q3">Como adicionar tempos em Python?</h1>
<p>Um colega do grupo perguntou como adicionar durações de tempo em Python.
Apresentando o seguinte código:</p>
<div class="highlight"><pre><code class="language-python"><span>from</span> datetime <span>import</span> time
t0 <span>=</span> time<span>.</span>fromisoformat(<span>'06:52:00'</span>)
t1 <span>=</span> time<span>.</span>fromisoformat(<span>'00:08:15'</span>)
t2 <span>=</span> time<span>.</span>fromisoformat(<span>'00:07:12'</span>)
t3 <span>=</span> t0 <span>+</span> t1 <span>+</span> t2
</code></pre></div><p>que resulta no erro seguinte:</p>
<pre><code>Traceback (most recent call last):
File "tdeltao.py", line 5, in <module>
t3 = t0 + t1 + t2
TypeError: unsupported operand type(s) for +: 'datetime.time' and 'datetime.time'
</code></pre><p>Isto acontece porque a classe <code>time</code> não define a operação de soma. A classe correta para este tipo de cálculo é <code>timedelta</code>.
Para calcular corretamente esta soma, precisamos primeiro converter a string em um objeto <code>timedelta</code>. Isto pode ser feito com uma função simples:</p>
<div class="highlight"><pre><code class="language-python"><span>from</span> datetime <span>import</span> time, timedelta, datetime
<span>def</span> <span>string_para_timedelta</span>(str_time: str) <span>-></span> timedelta:
valor <span>=</span> time<span>.</span>fromisoformat(str_time)
<span>return</span> timedelta(hours<span>=</span>valor<span>.</span>hour,
minutes<span>=</span>valor<span>.</span>minute,
seconds<span>=</span>valor<span>.</span>second)
t0 <span>=</span> string_para_timedelta(<span>'06:52:00'</span>)
t1 <span>=</span> string_para_timedelta(<span>'00:08:15'</span>)
t2 <span>=</span> string_para_timedelta(<span>'00:07:12'</span>)
t3 <span>=</span> t0 <span>+</span> t1 <span>+</span> t2
<span>print</span>(t3)
<span>print</span>(datetime<span>.</span>now() <span>+</span> t3)</code></pre></div>
<p>que resulta em:</p>
<pre><code>7:07:27
2020-05-03 22:48:55.473647
</code></pre><p>Uma dica para lembrar da diferença entre <code>time</code> e <code>timedelta</code> é que <code>time</code> não pode representar mais de 24h e <code>timedelta</code> pode representar mais de um século (270 anos). Outra vantagem de <code>timedelta</code> é poder ser utilizada em operações com <code>date</code> e <code>datetime</code>.</p>Nilo Ney Coutinho Menezeshttps://blog.nilo.pro.br/tags/python/Foco no ambiente, acelerando o aprendizado!https://avelino.run/foco-no-ambiente-acelerando-o-aprendizado/2020-02-28T16:49:00+00:00Objetivo desse blogpost é compartilhar como geralmente faço para acelerar meu aprendizado em uma área que não tenho tanta experiencia e quero (e/ou preciso) ganhar mais experiência.
Quando entrei na área de tecnologia (desenvolvimento de software) não sabia praticamente nada e comecei estudar como poderia acelerar meu aprendizado, até que me deparei em um texto no reddit que falava sobre foco no ambiente, foi o extremamente difícil eu conseguir entender o que estava querendo dizer aquele texto, mas depois de dias lendo e relendo consegui absorver que deveria frequentar lugares onde tinha pessoas fazendo o que buscava aprender, assim aceleraria meu aprendizado.<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/dke4ZftTa4o" width="1" />Thiago Avelinohttps://avelino.run/blog/Chegando no limite da tecnologia, e agora para aonde vou?https://avelino.run/chegando-no-limite-da-tecnologia-e-agora-para-aonde-vou/2020-02-19T14:00:00+00:00Nós de tecnologia em geral, somos early adopter (gostamos de abraçar novas tecnologias, mesmo sem saber ao certo porque ela existe), quando falamos em desenvolvimento não é muito diferente.
Por que não usamos o banco de dados X? Podemos usar a linguagem de programação Y! O serviço Z resolve 100% dos nossos problemas! Vamos assumir que as afirmações acima estejam 100% corretas (lançamos o primeiro erro), a solução irá servir para “vida toda” ou daqui a alguns meses tenham que olhar para ela, porque batemos em algum limite da implementação, arquitetura ou da própria tecnologia?<img alt="" height="1" src="http://feeds.feedburner.com/~r/pyavelino/~4/zVC2NcnDL6o" width="1" />Thiago Avelinohttps://avelino.run/blog/Criando um CI de uma aplicação Django usando Github Actionstag:pythonclub.com.br,2020-01-24:/django-ci-github-actions.html2020-01-24T15:10:00+00:00<p>Fala pessoal, tudo bom?</p>
<p>Nos vídeo abaixo vou mostrar como podemos configurar um CI de uma aplicação Django usando Github Actions.</p>
<p><a class="reference external" href="https://www.youtube.com/watch?v=KpSlY8leYFY">https://www.youtube.com/watch?v=KpSlY8leYFY</a>.</p>
<div class="youtube youtube-16x9"></div>Lucas Magnumhttp://pythonclub.com.br/Oxidizing sourmash: PR walkthroughtag:blog.luizirber.org,2020-01-10:/2020/01/10/sourmash-pr/2020-01-10T15:00:00+00:00<p>sourmash 3 was released last week,
finally landing the Rust backend.
But, what changes when developing new features in sourmash?
I was thinking about how to best document this process,
and since <a href="https://github.com/dib-lab/sourmash/pull/826/files">PR #826</a> is a short example touching all the layers I decided to do a
small walkthrough.</p>
<p>Shall we?</p>
<h2>The problem</h2>
<p>The first step is describing the problem,
and trying to convince reviewers (and yourself) that the changes bring enough benefits to justify a merge.
This is the description I put in the PR:</p>
<blockquote>
<p>Calling <code>.add_hash()</code> on a MinHash sketch is fine,
but if you're calling it all the time it's better to pass a list of hashes and call <code>.add_many()</code> instead.
Before this PR <code>add_many</code> just called <code>add_hash</code> for each hash it was passed,
but now it will pass the full list to Rust (and that's way faster).</p>
<p>No changes for public APIs,
and I changed the <code>_signatures</code> method in LCA to accumulate hashes for each sig first,
and then set them all at once.
This is way faster,
but might use more intermediate memory (I'll evaluate this now).</p>
</blockquote>
<p>There are many details that sound like jargon for someone not familiar with the codebase,
but if I write something too long I'll probably be wasting the reviewers time too.
The benefit of a very detailed description is extending the knowledge for other people
(not necessarily the maintainers),
but that also takes effort that might be better allocated to solve other problems.
Or, more realistically, putting out other fires =P</p>
<p>Nonetheless,
some points I like to add in PR descriptions:
- why is there a problem with the current approach?
- is this the minimal viable change, or is it trying to change too many things
at once? The former is way better, in general.
- what are the trade-offs? This PR is using more memory to lower the runtime,
but I hadn't measure it yet when I opened it.
- Not changing public APIs is always good to convince reviewers.
If the project follows a <a href="https://semver.org/">semantic versioning</a> scheme,
changes to the public APIs are major version bumps,
and that can brings other consequences for users.</p>
<h2>Setting up for changing code</h2>
<p>If this was a bug fix PR,
the first thing I would do is write a new test triggering the bug,
and then proceed to fix it in the code
(Hmm, maybe that would be another good walkthrough?).
But this PR is making performance claims ("it's going to be faster"),
and that's a bit hard to codify in tests.
<sup id="sf-sourmash-pr-1-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-1" title="We do have https://asv.readthedocs.io/ set up for micro-benchmarks, and now that I think about it... I could have started by writing a benchmark for add_many, and then showing that it is faster. I will add this approach to the sourmash PR checklist =]">1</a></sup>
Since it's also proposing to change a method (<code>_signatures</code> in LCA indices) that is better to benchmark with a real index (and not a toy example),
I used the same data and command I run in <a href="https://github.com/luizirber/sourmash_resources">sourmash_resources</a> to check how memory consumption and runtime changed.
For reference, this is the command: </p>
<div class="highlight"><pre><span></span>sourmash search -o out.csv --scaled <span class="m">2000</span> -k <span class="m">51</span> HSMA33OT.fastq.gz.sig genbank-k51.lca.json.gz
</pre></div>
<p>I'm using the <code>benchmark</code> feature from <a href="https://snakemake.readthedocs.io/">snakemake</a> in <a href="https://github.com/luizirber/sourmash_resources/blob/83ea237397d242e48c9d95eb0d9f50ceb4ad95c7/Snakefile#L99L114">sourmash_resources</a> to
track how much memory, runtime and I/O is used for each command (and version) of sourmash,
and generate the plots in the README in that repo.
That is fine for a high-level view ("what's the maximum memory used?"),
but not so useful for digging into details ("what method is consuming most memory?").</p>
<p>Another additional problem is the dual<sup id="sf-sourmash-pr-2-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-2" title="or triple, if you count C">2</a></sup> language nature of sourmash,
where we have Python calling into Rust code (via CFFI).
There are great tools for measuring and profiling Python code,
but they tend to not work with extension code...</p>
<p>So, let's bring two of my favorite tools to help!</p>
<h3>Memory profiling: heaptrack</h3>
<p><a href="https://github.com/KDE/heaptrack">heaptrack</a> is a heap profiler, and I first heard about it from <a href="https://www.vincentprouillet.com/">Vincent Prouillet</a>.
Its main advantage over other solutions (like valgrind's massif) is the low
overhead and... how easy it is to use:
just stick <code>heaptrack</code> in front of your command,
and you're good to go!</p>
<p>Example output:</p>
<div class="highlight"><pre><span></span>$ heaptrack sourmash search -o out.csv --scaled <span class="m">2000</span> -k <span class="m">51</span> HSMA33OT.fastq.gz.sig genbank-k51.lca.json.gz
heaptrack stats:
allocations: <span class="m">1379353</span>
leaked allocations: <span class="m">1660</span>
temporary allocations: <span class="m">168984</span>
Heaptrack finished! Now run the following to investigate the data:
heaptrack --analyze heaptrack.sourmash.66565.gz
</pre></div>
<p><code>heaptrack --analyze</code> is a very nice graphical interface for analyzing the results,
but for this PR I'm mostly focusing on the Summary page (and overall memory consumption).
Tracking allocations in Python doesn't give many details,
because it shows the CPython functions being called,
but the ability to track into the extension code (Rust) allocations is amazing
for finding bottlenecks (and memory leaks =P).
<sup id="sf-sourmash-pr-3-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-3" title="It would be super cool to have the unwinding code from py-spy in heaptrack, and be able to see exactly what Python methods/lines of code were calling the Rust parts...">3</a></sup></p>
<h3>CPU profiling: py-spy</h3>
<p>Just as other solutions exist for profiling memory,
there are many for profiling CPU usage in Python,
including <code>profile</code> and <code>cProfile</code> in the standard library.
Again, the issue is being able to analyze extension code,
and bringing the cannon (the <code>perf</code> command in Linux, for example) looses the
benefit of tracking Python code properly (because we get back the CPython
functions, not what you defined in your Python code).</p>
<p>Enters <a href="https://github.com/benfred/py-spy">py-spy</a> by <a href="https://www.benfrederickson.com">Ben Frederickson</a>,
based on the <a href="https://github.com/rbspy/rbspy">rbspy</a> project by <a href="https://jvns.ca">Julia Evans</a>.
Both use a great idea:
read the process maps for the interpreters and resolve the full stack trace information,
with low overhead (because it uses sampling).
<a href="https://github.com/benfred/py-spy">py-spy</a> also goes further and resolves <a href="https://www.benfrederickson.com/profiling-native-python-extensions-with-py-spy/">native Python extensions</a> stack traces,
meaning we can get the complete picture all the way from the Python CLI to the
Rust core library!<sup id="sf-sourmash-pr-4-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-4" title="Even if py-spy doesn't talk explicitly about Rust, it works very very well, woohoo!">4</a></sup></p>
<p><code>py-spy</code> is also easy to use:
stick <code>py-spy record --output search.svg -n --</code> in front of the command,
and it will generate a flamegraph in <code>search.svg</code>.
The full command for this PR is</p>
<div class="highlight"><pre><span></span>py-spy record --output search.svg -n -- sourmash search -o out.csv --scaled <span class="m">2000</span> -k <span class="m">51</span> HSMA.fastq.sig genbank-k51.lca.json.gz
</pre></div>
<h2>Show me the code!</h2>
<p>OK, OK, sheesh. But it's worth repeating: the code is important, but there are
many other aspects that are just as important =]</p>
<h3>Replacing <code>add_hash</code> calls with one <code>add_many</code></h3>
<p>Let's start at the <a href="https://github.com/dib-lab/sourmash/pull/826/files#diff-adf06d14c535d5b22da9fb3862e4a487"><code>_signatures()</code></a> method on LCA indices.
This is the original method:</p>
<div class="highlight"><pre><span></span><span class="nd">@cached_property</span>
<span class="k">def</span> <span class="nf">_signatures</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
<span class="s2">"Create a _signatures member dictionary that contains {idx: minhash}."</span>
<span class="kn">from</span> <span class="nn">..</span> <span class="kn">import</span> <span class="n">MinHash</span>
<span class="n">minhash</span> <span class="o">=</span> <span class="n">MinHash</span><span class="p">(</span><span class="n">n</span><span class="o">=</span><span class="mi">0</span><span class="p">,</span> <span class="n">ksize</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">ksize</span><span class="p">,</span> <span class="n">scaled</span><span class="o">=</span><span class="bp">self</span><span class="o">.</span><span class="n">scaled</span><span class="p">)</span>
<span class="n">debug</span><span class="p">(</span><span class="s1">'creating signatures for LCA DB...'</span><span class="p">)</span>
<span class="n">sigd</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="n">minhash</span><span class="o">.</span><span class="n">copy_and_clear</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hashval_to_idx</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">vv</span> <span class="ow">in</span> <span class="n">v</span><span class="p">:</span>
<span class="n">sigd</span><span class="p">[</span><span class="n">vv</span><span class="p">]</span><span class="o">.</span><span class="n">add_hash</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
<span class="n">debug</span><span class="p">(</span><span class="s1">'=> </span><span class="si">{}</span><span class="s1"> signatures!'</span><span class="p">,</span> <span class="nb">len</span><span class="p">(</span><span class="n">sigd</span><span class="p">))</span>
<span class="k">return</span> <span class="n">sigd</span>
</pre></div>
<p><code>sigd[vv].add_hash(k)</code> is the culprit.
Each call to <code>.add_hash</code> has to go thru CFFI to reach the extension code,
and the overhead is significant.
It is a similar situation to accessing array elements in NumPy:
it works,
but it is way slower than using operations that avoid crossing from Python to
the extension code.
What we want to do instead is call <code>.add_many(hashes)</code>,
which takes a list of hashes and process it entirely in Rust
(ideally. We will get there).</p>
<p>But, to have a list of hashes, there is another issue with this code.</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hashval_to_idx</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">vv</span> <span class="ow">in</span> <span class="n">v</span><span class="p">:</span>
<span class="n">sigd</span><span class="p">[</span><span class="n">vv</span><span class="p">]</span><span class="o">.</span><span class="n">add_hash</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
</pre></div>
<p>There are two nested for loops,
and <code>add_hash</code> is being called with values from the inner loop.
So... we don't have the list of hashes beforehand.</p>
<p>But we can change the code a bit to save the hashes for each signature
in a temporary list,
and then call <code>add_many</code> on the temporary list.
Like this:</p>
<div class="highlight"><pre><span></span><span class="n">temp_vals</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">list</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hashval_to_idx</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">vv</span> <span class="ow">in</span> <span class="n">v</span><span class="p">:</span>
<span class="n">temp_vals</span><span class="p">[</span><span class="n">vv</span><span class="p">]</span><span class="o">.</span><span class="n">append</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
<span class="k">for</span> <span class="n">sig</span><span class="p">,</span> <span class="n">vals</span> <span class="ow">in</span> <span class="n">temp_vals</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">sigd</span><span class="p">[</span><span class="n">sig</span><span class="p">]</span><span class="o">.</span><span class="n">add_many</span><span class="p">(</span><span class="n">vals</span><span class="p">)</span>
</pre></div>
<p>There is a trade-off here:
if we save the hashes in temporary lists,
will the memory consumption be so high that the runtime gains of calling
<code>add_many</code> in these temporary lists be cancelled?</p>
<p>Time to measure it =]</p>
<table>
<thead>
<tr>
<th align="left">version</th>
<th align="left">mem</th>
<th align="left">time</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">original</td>
<td align="left">1.5 GB</td>
<td align="left">160s</td>
</tr>
<tr>
<td align="left"><code>list</code></td>
<td align="left">1.7GB</td>
<td align="left">173s</td>
</tr>
</tbody>
</table>
<p>Wait, it got worse?!?! Building temporary lists only takes time and memory,
and bring no benefits!</p>
<p>This mystery goes away when you look at the <a href="https://github.com/dib-lab/sourmash/pull/826/files#diff-2f53b2a5be4083c39a0275847c87f88fR190">add_many method</a>:</p>
<div class="highlight"><pre><span></span><span class="k">def</span> <span class="nf">add_many</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">hashes</span><span class="p">):</span>
<span class="s2">"Add many hashes in at once."</span>
<span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">hashes</span><span class="p">,</span> <span class="n">MinHash</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_methodcall</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">kmerminhash_add_from</span><span class="p">,</span> <span class="n">hashes</span><span class="o">.</span><span class="n">_objptr</span><span class="p">)</span>
<span class="k">else</span><span class="p">:</span>
<span class="k">for</span> <span class="nb">hash</span> <span class="ow">in</span> <span class="n">hashes</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_methodcall</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">kmerminhash_add_hash</span><span class="p">,</span> <span class="nb">hash</span><span class="p">)</span>
</pre></div>
<p>The first check in the <code>if</code> statement is a shortcut for adding hashes from
another <code>MinHash</code>, so let's focus on <code>else</code> part...
And turns out that <code>add_many</code> is lying!
It doesn't process the <code>hashes</code> in the Rust extension,
but just loops and call <code>add_hash</code> for each <code>hash</code> in the list.
That's not going to be any faster than what we were doing in <code>_signatures</code>.</p>
<p>Time to fix <code>add_many</code>!</p>
<h3>Oxidizing <code>add_many</code></h3>
<p>The idea is to change this loop in <code>add_many</code>:</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="nb">hash</span> <span class="ow">in</span> <span class="n">hashes</span><span class="p">:</span>
<span class="bp">self</span><span class="o">.</span><span class="n">_methodcall</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">kmerminhash_add_hash</span><span class="p">,</span> <span class="nb">hash</span><span class="p">)</span>
</pre></div>
<p>with a call to a Rust extension function:</p>
<div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">_methodcall</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">kmerminhash_add_many</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="n">hashes</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">hashes</span><span class="p">))</span>
</pre></div>
<p><code>self._methodcall</code> is a convenience method defined in <a href="https://github.com/dib-lab/sourmash/blob/c6cbdf0398ef836797492e13371a38373c544ae1/sourmash/utils.py#L24">RustObject</a>
which translates a method-like call into a function call,
since our C layer only has functions.
This is the C prototype for this function:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span> <span class="nf">kmerminhash_add_many</span><span class="p">(</span>
<span class="n">KmerMinHash</span> <span class="o">*</span><span class="n">ptr</span><span class="p">,</span>
<span class="k">const</span> <span class="kt">uint64_t</span> <span class="o">*</span><span class="n">hashes_ptr</span><span class="p">,</span>
<span class="kt">uintptr_t</span> <span class="n">insize</span>
<span class="p">);</span>
</pre></div>
<p>You can almost read it as a Python method declaration,
where <code>KmerMinHash *ptr</code> means the same as the <code>self</code> in Python methods.
The other two arguments are a common idiom when passing pointers to data in C,
with <code>insize</code> being how many elements we have in the list.
<sup id="sf-sourmash-pr-5-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-5" title="Let's not talk about lack of array bounds checks in C...">5</a></sup>.
<code>CFFI</code> is very good at converting Python lists into pointers of a specific type,
as long as the type is of a primitive type
(<code>uint64_t</code> in our case, since each hash is a 64-bit unsigned integer number).</p>
<p>And the Rust code with the implementation of the function:</p>
<div class="highlight"><pre><span></span><span class="n">ffi_fn</span><span class="o">!</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="k">unsafe</span><span class="w"> </span><span class="k">fn</span> <span class="nf">kmerminhash_add_many</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ptr</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">KmerMinHash</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">hashes_ptr</span>: <span class="o">*</span><span class="k">const</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">insize</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">mh</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">assert</span><span class="o">!</span><span class="p">(</span><span class="o">!</span><span class="n">ptr</span><span class="p">.</span><span class="n">is_null</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="o">*</span><span class="n">ptr</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="kd">let</span><span class="w"> </span><span class="n">hashes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">assert</span><span class="o">!</span><span class="p">(</span><span class="o">!</span><span class="n">hashes_ptr</span><span class="p">.</span><span class="n">is_null</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">slice</span>::<span class="n">from_raw_parts</span><span class="p">(</span><span class="n">hashes_ptr</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">insize</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">hashes</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">mh</span><span class="p">.</span><span class="n">add_hash</span><span class="p">(</span><span class="o">*</span><span class="n">hash</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
</pre></div>
<p>Let's break what's happening here into smaller pieces.
Starting with the function signature:</p>
<div class="highlight"><pre><span></span><span class="n">ffi_fn</span><span class="o">!</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="k">unsafe</span><span class="w"> </span><span class="k">fn</span> <span class="nf">kmerminhash_add_many</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ptr</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">KmerMinHash</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">hashes_ptr</span>: <span class="o">*</span><span class="k">const</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">insize</span>: <span class="kt">usize</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="p">)</span><span class="w"> </span>-> <span class="nb">Result</span><span class="o"><</span><span class="p">()</span><span class="o">></span><span class="w"></span>
</pre></div>
<p>The weird <code>ffi_fn! {}</code> syntax around the function is a macro in Rust:
it changes the final generated code to convert the return value (<code>Result<()></code>) into something that is valid C code (in this case, <code>void</code>).
What happens if there is an error, then?
The Rust extension has code for passing back an error code and message to Python,
as well as capturing panics (when things go horrible bad and the program can't recover)
in a way that Python can then deal with (raising exceptions and cleaning up).
It also sets the <code>#[no_mangle]</code> attribute in the function,
meaning that the final name of the function will follow C semantics (instead of Rust semantics),
and can be called more easily from C and other languages.
This <code>ffi_fn!</code> macro comes from <a href="https://github.com/getsentry/symbolic">symbolic</a>,
a big influence on the design of the Python/Rust bridge in sourmash.</p>
<p><code>unsafe</code> is the keyword in Rust to disable some checks in the code to allow
potentially dangerous things (like dereferencing a pointer),
and it is required to interact with C code.
<code>unsafe</code> doesn't mean that the code is always unsafe to use:
it's up to whoever is calling this to verify that valid data is being passed and invariants are being preserved.</p>
<p>If we remove the <code>ffi_fn!</code> macro and the <code>unsafe</code> keyword,
we have</p>
<div class="highlight"><pre><span></span><span class="k">fn</span> <span class="nf">kmerminhash_add_many</span><span class="p">(</span><span class="w"></span>
<span class="w"> </span><span class="n">ptr</span>: <span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="n">KmerMinHash</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">hashes_ptr</span>: <span class="o">*</span><span class="k">const</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="n">insize</span>: <span class="kt">usize</span>
<span class="p">);</span><span class="w"></span>
</pre></div>
<p>At this point we can pretty much map between Rust and the C function prototype:</p>
<div class="highlight"><pre><span></span><span class="kt">void</span> <span class="nf">kmerminhash_add_many</span><span class="p">(</span>
<span class="n">KmerMinHash</span> <span class="o">*</span><span class="n">ptr</span><span class="p">,</span>
<span class="k">const</span> <span class="kt">uint64_t</span> <span class="o">*</span><span class="n">hashes_ptr</span><span class="p">,</span>
<span class="kt">uintptr_t</span> <span class="n">insize</span>
<span class="p">);</span>
</pre></div>
<p>Some interesting points:</p>
<ul>
<li>We use <code>fn</code> to declare a function in Rust.</li>
<li>The type of an argument comes after the name of the argument in Rust,
while it's the other way around in C.
Same for the return type (it is omitted in the Rust function, which means it
is <code>-> ()</code>, equivalent to a <code>void</code> return type in C).</li>
<li>In Rust everything is <strong>immutable</strong> by default, so we need to say that we want
a mutable pointer to a <code>KmerMinHash</code> item: <code>*mut KmerMinHash</code>).
In C everything is mutable by default.</li>
<li><code>u64</code> in Rust -> <code>uint64_t</code> in C</li>
<li><code>usize</code> in Rust -> <code>uintptr_t</code> in C</li>
</ul>
<p>Let's check the implementation of the function now.
We start by converting the <code>ptr</code> argument (a raw pointer to a <code>KmerMinHash</code> struct)
into a regular Rust struct:</p>
<div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">mh</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">assert</span><span class="o">!</span><span class="p">(</span><span class="o">!</span><span class="n">ptr</span><span class="p">.</span><span class="n">is_null</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="o">&</span><span class="k">mut</span><span class="w"> </span><span class="o">*</span><span class="n">ptr</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</pre></div>
<p>This block is asserting that <code>ptr</code> is not a null pointer,
and if so it dereferences it and store in a mutable reference.
If it was a null pointer the <code>assert!</code> would panic (which might sound extreme,
but is way better than continue running because dereferencing a null pointer is
BAD).
Note that functions always need all the types in arguments and return values,
but for variables in the body of the function
Rust can figure out types most of the time,
so no need to specify them.</p>
<p>The next block prepares our list of hashes for use:</p>
<div class="highlight"><pre><span></span><span class="kd">let</span><span class="w"> </span><span class="n">hashes</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">assert</span><span class="o">!</span><span class="p">(</span><span class="o">!</span><span class="n">hashes_ptr</span><span class="p">.</span><span class="n">is_null</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">slice</span>::<span class="n">from_raw_parts</span><span class="p">(</span><span class="n">hashes_ptr</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="o">*</span><span class="k">mut</span><span class="w"> </span><span class="kt">u64</span><span class="p">,</span><span class="w"> </span><span class="n">insize</span><span class="p">)</span><span class="w"></span>
<span class="p">};</span><span class="w"></span>
</pre></div>
<p>We are again asserting that the <code>hashes_ptr</code> is not a null pointer,
but instead of dereferencing the pointer like before we use it to create a <code>slice</code>,
a dynamically-sized view into a contiguous sequence.
The list we got from Python is a contiguous sequence of size <code>insize</code>,
and the <code>slice::from_raw_parts</code> function creates a slice from a pointer to data and a size.</p>
<p>Oh, and can you spot the bug?
I created the slice using <code>*mut u64</code>,
but the data is declared as <code>*const u64</code>.
Because we are in an <code>unsafe</code> block Rust let me change the mutability,
but I shouldn't be doing that,
since we don't need to mutate the slice.
Oops.</p>
<p>Finally, let's add hashes to our MinHash!
We need a <code>for</code> loop, and call <code>add_hash</code> for each <code>hash</code>:</p>
<div class="highlight"><pre><span></span><span class="k">for</span><span class="w"> </span><span class="n">hash</span><span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="n">hashes</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">mh</span><span class="p">.</span><span class="n">add_hash</span><span class="p">(</span><span class="o">*</span><span class="n">hash</span><span class="p">);</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>
<span class="nb">Ok</span><span class="p">(())</span><span class="w"></span>
</pre></div>
<p>We finish the function with <code>Ok(())</code> to indicate no errors occurred.</p>
<p>Why is calling <code>add_hash</code> here faster than what we were doing before in Python?
Rust can optimize these calls and generate very efficient native code,
while Python is an interpreted language and most of the time don't have the same
guarantees that Rust can leverage to generate the code.
And, again,
calling <code>add_hash</code> here doesn't need to cross FFI boundaries or,
in fact,
do any dynamic evaluation during runtime,
because it is all statically analyzed during compilation.</p>
<h2>Putting it all together</h2>
<p>And... that's the PR code.
There are some other unrelated changes that should have been in new PRs,
but since they were so small it would be more work than necessary.
OK, that's a lame excuse:
it's confusing for reviewers to see these changes here,
so avoid doing that if possible!</p>
<p>But, did it work?</p>
<table>
<thead>
<tr>
<th align="left">version</th>
<th align="left">mem</th>
<th align="left">time</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">original</td>
<td align="left">1.5 GB</td>
<td align="left">160s</td>
</tr>
<tr>
<td align="left"><code>list</code></td>
<td align="left">1.7GB</td>
<td align="left">73s</td>
</tr>
</tbody>
</table>
<p>We are using 200 MB of extra memory,
but taking less than half the time it was taking before.
I think this is a good trade-off,
and so did the <a href="https://github.com/dib-lab/sourmash/pull/826#pullrequestreview-339020803">reviewer</a> and the PR was approved.</p>
<p>Hopefully this was useful, 'til next time!</p>
<h2>Comments?</h2>
<ul>
<li><a href="https://social.lasanha.org/@luizirber/103461534713587975">Thread on Mastodon</a></li>
<li><a href="https://twitter.com/luizirber/status/1215772245928235008">Thread on Twitter</a></li>
<li><a href="https://lobste.rs/s/xnaugq/oxidizing_sourmash_pr_walkthrough">Lobste.rs submission</a></li>
</ul>
<h2>Bonus: <code>list</code> or <code>set</code>?</h2>
<p>The first version of the PR used a <code>set</code> instead of a <code>list</code> to accumulate hashes.
Since a <code>set</code> doesn't have repeated elements,
this could potentially use less memory.
The code:</p>
<div class="highlight"><pre><span></span><span class="n">temp_vals</span> <span class="o">=</span> <span class="n">defaultdict</span><span class="p">(</span><span class="nb">set</span><span class="p">)</span>
<span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hashval_to_idx</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">vv</span> <span class="ow">in</span> <span class="n">v</span><span class="p">:</span>
<span class="n">temp_vals</span><span class="p">[</span><span class="n">vv</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
<span class="k">for</span> <span class="n">sig</span><span class="p">,</span> <span class="n">vals</span> <span class="ow">in</span> <span class="n">temp_vals</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="n">sigd</span><span class="p">[</span><span class="n">sig</span><span class="p">]</span><span class="o">.</span><span class="n">add_many</span><span class="p">(</span><span class="n">vals</span><span class="p">)</span>
</pre></div>
<p>The runtime was again half of the original,
but...</p>
<table>
<thead>
<tr>
<th align="left">version</th>
<th align="left">mem</th>
<th align="left">time</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">original</td>
<td align="left">1.5 GB</td>
<td align="left">160s</td>
</tr>
<tr>
<td align="left"><code>set</code></td>
<td align="left">3.8GB</td>
<td align="left">80s</td>
</tr>
<tr>
<td align="left"><code>list</code></td>
<td align="left">1.7GB</td>
<td align="left">73s</td>
</tr>
</tbody>
</table>
<p>... memory consumption was almost 2.5 times the original! WAT</p>
<p>The culprit this time? The new <code>kmerminhash_add_many</code> call in the <code>add_many</code>
method.
This one:</p>
<div class="highlight"><pre><span></span><span class="bp">self</span><span class="o">.</span><span class="n">_methodcall</span><span class="p">(</span><span class="n">lib</span><span class="o">.</span><span class="n">kmerminhash_add_many</span><span class="p">,</span> <span class="nb">list</span><span class="p">(</span><span class="n">hashes</span><span class="p">),</span> <span class="nb">len</span><span class="p">(</span><span class="n">hashes</span><span class="p">))</span>
</pre></div>
<p><code>CFFI</code> doesn't know how to convert a <code>set</code> into something that C understands,
so we need to call <code>list(hashes)</code> to convert it into a list.
Since Python (and <code>CFFI</code>) can't know if the data is going to be used later
<sup id="sf-sourmash-pr-6-back"><a class="simple-footnote" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-6" title="something that the memory ownership model in Rust does, BTW">6</a></sup>
it needs to keep it around
(and be eventually deallocated by the garbage collector).
And that's how we get at least double the memory being allocated...</p>
<p>There is another lesson here.
If we look at the <code>for</code> loop again:</p>
<div class="highlight"><pre><span></span><span class="k">for</span> <span class="p">(</span><span class="n">k</span><span class="p">,</span> <span class="n">v</span><span class="p">)</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">hashval_to_idx</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
<span class="k">for</span> <span class="n">vv</span> <span class="ow">in</span> <span class="n">v</span><span class="p">:</span>
<span class="n">temp_vals</span><span class="p">[</span><span class="n">vv</span><span class="p">]</span><span class="o">.</span><span class="n">add</span><span class="p">(</span><span class="n">k</span><span class="p">)</span>
</pre></div>
<p>each <code>k</code> is already unique because they are keys in the <code>hashval_to_idx</code> dictionary,
so the initial assumption
(that a <code>set</code> might save memory because it doesn't have repeated elements)
is... irrelevant for the problem =]</p><hr /><h2>Footnotes</h2><ol><li id="sf-sourmash-pr-1"><p>We do have https://asv.readthedocs.io/ set up for micro-benchmarks,
and now that I think about it...
I could have started by writing a benchmark for <code>add_many</code>,
and then showing that it is faster.
I will add this approach to the sourmash PR checklist =] <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-1-back">↩</a></p></li><li id="sf-sourmash-pr-2"><p>or triple, if you count C <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-2-back">↩</a></p></li><li id="sf-sourmash-pr-3"><p>It would be super cool to have the unwinding code from py-spy in heaptrack,
and be able to see exactly what Python methods/lines of code were calling the
Rust parts... <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-3-back">↩</a></p></li><li id="sf-sourmash-pr-4"><p>Even if py-spy doesn't talk explicitly about Rust,
it works very very well, woohoo! <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-4-back">↩</a></p></li><li id="sf-sourmash-pr-5"><p>Let's not talk about lack of array bounds checks in C... <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-5-back">↩</a></p></li><li id="sf-sourmash-pr-6"><p>something that the memory ownership model in Rust does, BTW <a class="simple-footnote-back" href="https://blog.luizirber.org/atom.xml?tag=python#sf-sourmash-pr-6-back">↩</a></p></li></ol>luizirberhttps://blog.luizirber.org/Try out Tsuru: announcing limited previewtag:blogger.com,1999:blog-4820627057316560708.post-54894687705611916352019-12-10T03:42:44+00:00A few days ago, <a href="http://tsuru.io/" target="_blank">Tsuru</a> got some <a href="https://news.ycombinator.com/item?id=5436648" rel="nofollow" target="_blank">attention</a> in the <a href="http://www.wired.com/wiredenterprise/2013/03/tsuru/" rel="nofollow" target="_blank">news</a>. After reading about Tsuru, and seeing some of its capabilities, people started asking for a way to try Tsuru. Well, your claims were attended! We're preparing a public cloud that will be freely available for beta testers.<br /><br /><b>TL;DR:</b> go to <a href="http://www.tsuru.io/try" target="_blank">tsuru.io/try</a>, signup for beta testing and get ready to start deploying Python, Ruby, Go and Java applications in the cloud.<br /><a name="more"></a><br /><h2>What is Tsuru?</h2>Tsuru is an <a href="https://github.com/globocom/tsuru" target="_blank">open source</a> platform as a service that allows developers to automatically deploy and manage web applications written in many different platforms (like <a href="http://python.org/" target="_blank">Python</a>, <a href="http://ruby-lang.org/" target="_blank">Ruby</a> and <a href="http://golang.org/" target="_blank">Go</a>). It aims to provide a solution for cloud computing platforms that is extensible, flexible and component based.<br /><br />You can run your own public or private cloud using Tsuru. Or you can try it in the public cloud that Globo.com is building.<br /><br /><h2>What is Tsuru public cloud? What does "beta availability" means?</h2>Tsuru public cloud will be a public, freely available, installation of Tsuru, provided by <a href="http://globo.com/" target="_blank">Globo.com</a>. "Beta availability" means that it will not be available for the general Internet public.<br /><br />People will need to subscribe for the beta testing and wait for the confirmation, so they can start deploying web applications on Tsuru public cloud.<br /><br /><h2>Which development platforms are going to be available?</h2>Tsuru already supports Ruby, Python, Java and Go, so it is very likely that these platforms will be available for all beta users.<br /><br />It's important to notice that adding new platforms to Tsuru is a straightforward task: each development platform is based on <a href="http://jujucharms.com/" rel="nofollow" target="_blank">Juju Charms</a>, so one can adapt charms available at Charm Store and <a href="https://github.com/globocom/charms" target="_blank">send a patch</a>.<br /><br /><h2>How limited is it going to be?</h2>We don't know what's the proper answer for this question yet, but don't worry about numbers now. There will be some kind of per-user quota, but it has not been defined yet.<br /><br />People interested in running applications in the Tsuru public cloud that get to use the beta version will have access a functional environment where they will be able to deploy <i>at least</i> one web application.<br /><br /><h2>When will it be available?</h2>We're <a href="https://github.com/globocom/tsuru/issues" target="_blank">working hard</a> to make it available as soon as possible, and you can help us get it done! If you want to contribute, please take a look at <a href="https://github.com/globocom/tsuru" target="_blank">Tsuru repository</a>, chose an issue, <a href="http://www.tsuru.io/community" rel="nofollow" target="_blank">discuss</a> your solution and send your patches. We are going to be very happy helping you out.<br /><br /><h2>What if I don't want to wait?</h2>If you want an unlimited, fully manageable and customized installation of Tsuru, you can have it today. Check out Tsuru's <a href="http://docs.tsuru.io/" target="_blank">documentation</a> and, in case of doubts, don't hesitate in contacting the newborn <a href="http://www.tsuru.io/community" target="_blank">Tsuru community</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Setting up a Django production environment: compiling and configuring nginxtag:blogger.com,1999:blog-4820627057316560708.post-18679948947697289952019-12-10T03:42:44+00:00<img align="right" border="0" height="92" src="https://4.bp.blogspot.com/-FbHIACBZ8yw/T_7WxpAJI-I/AAAAAAAAA8I/CYdO1Xh4aLc/s400/nginx-logo.png" width="295" />Here is another series of posts: now I’m going to write about setting up a <a href="http://djangoproject.com" rel="nofollow" target="_blank">Django</a> production environment using <a href="http://nginx.org" rel="nofollow" target="_blank">nginx</a> and <a href="http://gunicorn.org" rel="nofollow" target="_blank">Green Unicorn</a> in a <a href="http://virtualenv.org" rel="nofollow" target="_blank">virtual environment</a>. The subject in this first post is nginx, which is my favorite web server. <br /><br />This post explains how to install nginx from sources, compiling it (on Linux). You might want to use <em>apt</em>, <em>zif</em>, <em>yum</em> or <em>ports</em>, but I prefer building from sources. So, to build from sources, make sure you have all development dependencies (C headers, including the PCRE library headers, nginx rewrite module uses it). If you want to build nginx with SSL support, keep in mind that you will need the libssl headers too.<a name="more"></a><br /><br />Build nginx from source is a straightforward process: all you need to do is download it from the official site and build with some simple options. In our setup, we’re going to install nginx under <code>/opt/nginx</code>, and use it with the nginx system user. So, let’s download and extract the latest stable version (1.0.9) from nginx website: <pre><br />% curl -O http://nginx.org/download/nginx-1.0.9.tar.gz<br />% tar -xzf nginx-1.0.9.tar.gz<br /></pre>Once you have extracted it, just configure, compile and install: <pre><br />% ./configure --prefix=/opt/nginx --user=nginx --group=nginx<br />% make<br />% [sudo] make install<br /></pre>As you can see, we provided the <code>/opt/nginx</code> to configure, make sure the <code>/opt</code> directory exists. Also, make sure that there is a user and a group called <em>nginx</em>, if they don’t exist, add them: <pre>% [sudo] adduser --system --no-create-home --disabled-login --disabled-password --group nginx</pre>After that, you can start nginx using the command line below: <pre>% [sudo] /opt/nginx/sbin/nginx</pre><blockquote> <p>Linode provides an <a href="http://library.linode.com/assets/634-init-deb.sh" rel="nofollow" target="_blank">init script</a> that uses <a href="http://man.he.net/man8/start-stop-daemon" rel="nofollow" target="_blank">start-stop-daemon</a>, you might want to use it.</p></blockquote><h4>nginx configuration</h4>nginx comes with a default <code>nginx.conf</code> file, let’s change it to reflect the following configuration requirements: <ul> <li>nginx should start workers with the <code>nginx</code> user</li> <li>nginx should have two worker processes</li> <li>the PID should be stored in the <code>/opt/nginx/log/nginx.pid</code> file</li> <li>nginx must have an access log in <code>/opt/nginx/logs/access.log</code></li> <li>the configuration for the Django project we’re going to develop should be versioned with the entire code, so it must be included in the <code>nginx.conf</code> file (assume that the <em>library</em> project is in the directory <code>/opt/projects</code>).</li></ul>So here is the <code>nginx.conf</code> for the requirements above: <pre><br />user nginx;<br />worker_processes 2;<br /><br />pid logs/nginx.pid;<br /><br />events {<br /> worker_connections 1024;<br />}<br /><br />http {<br /> include mime.types;<br /> default_type application/octet-stream;<br /><br /> log_format main '$remote_addr - $remote_user [$time_local] "$request" '<br /> '$status $body_bytes_sent "$http_referer" '<br /> '"$http_user_agent" "$http_x_forwarded_for"';<br /><br /> access_log logs/access.log main;<br /><br /> sendfile on;<br /> keepalive_timeout 65;<br /><br /> include /opt/projects/showcase/nginx.conf;<br />}<br /></pre>Now we just need to write the configuration for our Django project. I’m using an old sample project written while I was working at <a href="http://www.giran.com.br" rel="nofollow" target="_blank">Giran</a>: the name is <em>lojas giranianas</em>, a nonsense portuguese joke with a famous brazilian store. It’s an unfinished showcase of products, it’s like an e-commerce project, but it can’t sell, so it’s just a product catalog. The code is available at <a href="https://github.com/fsouza/fast-track-django" rel="nofollow" target="_blank">Github</a>. The <code>nginx.conf</code> file for the repository is here: <pre><br />server {<br /> listen 80;<br /> server_name localhost;<br /><br /> charset utf-8;<br /><br /> location / {<br /> proxy_set_header X-Real-IP $remote_addr;<br /> proxy_set_header Host $http_host;<br /> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br /><br /> proxy_pass http://localhost:8000;<br /> }<br /><br /> location /static {<br /> root /opt/projects/showcase/;<br /> expires 1d;<br /> }<br />}<br /></pre>The server listens on port <code>80</code>, responds for the <code>localhost</code> hostname (<a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html" target="_blank">read more about the Host header</a>). The <code>location /static</code> directive says that nginx will serve the static files of the project. It also includes an <code>expires</code> directive for caching control. The <code>location /</code> directive makes a <code>proxy_pass</code>, forwarding all requisitions to an upstream server listening on port 8000, this server is the subject of the next post of the series: the Green Unicorn (gunicorn) server. <br /><br />Not only the HTTP request itself is forwarded to the gunicorn server, but also some headers, that helps to properly deal with the request: <ul><li><strong>X-Real-IP:</strong> forwards the remote address to the upstream server, so it can know the real IP of the user. When nginx forwards the request to gunicorn, without this header, all gunicorn will know is that there is a request coming from localhost (or wherever the nginx server is), the remote address is always the IP address of the machine where nginx is running (who actually make the request to gunicorn)</li><li><strong>Host:</strong> the <code>Host</code> header is forwarded so gunicorn can treat different requests for different hosts. Without this header, it will be impossible to Gunicorn to have these constraints</li><li><strong>X-Forwarded-For:</strong> also known as XFF, this header provide more precise information about the real IP who makes the request. Imagine there are 10 proxies between the user machine and your webserver, the XFF header will all these proxies comma separated. In order to not turn a proxy into an anonymizer, it’s a good practice to always forward this header.</li></ul>So that is it, in the next post we are going to install and run gunicorn. In other posts, we’ll see how to make automated deploys using <a href="http://fabfile.org" rel="nofollow" target="_blank">Fabric</a>, and some tricks on caching (using the <code>proxy_cache</code> directive and integrating <a href="http://f.souza.cc/search/label/django">Django</a>, nginx and <a href="http://memcached.org" rel="nofollow" target="_blank">memcached</a>). <br /><br />See you in next posts.fsouzanoreply@blogger.comhttp://f.souza.cc/Speaking at OSCON 2014tag:blogger.com,1999:blog-4820627057316560708.post-30174596073053722802019-12-10T03:42:28+00:00Wow, one year without any posts! But I'm trying to get back... <br /><br />This is a very short post, just to tell everybody that this year, I will have the opportunity to speak at OSCON 2014. I'm speaking about <a href="http://tsuru.io/" target="_blank">tsuru</a>, and check more details of the talk in the <a href="http://blog.tsuru.io/2014/04/11/tsuru-at-oscon-2014/" target="_blank">tsuru blog</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Creating HTML 5 slide presentations using landslidetag:blogger.com,1999:blog-4820627057316560708.post-33334988360237530242019-12-10T03:42:20+00:00<img align="right" border="0" height="156" src="http://1.bp.blogspot.com/-MXAwFibbZvE/T_5FJIB0PhI/AAAAAAAAA7c/AM3OYpF8QRU/s400/slide.jpg" width="250" />Recently I found <a href="https://github.com/adamzap/landslide" rel="nofollow" target="_blank">landslide</a>, which is a <a href="http://python.org" rel="nofollow" target="_blank">Python</a> tool for creating HTML 5 slide presentations. <br /><br />It’s based in a <a href="http://slides.html5rocks.com/" rel="nofollow" target="_blank">famous slide presentation</a>. It’s a simple script that generates HTML from a source file, which can be formatted using <a href="http://docutils.sourceforge.net/docs/ref/rst/introduction.html" rel="nofollow" target="_blank">reStructuredText</a>, <a href="http://www.textism.com/tools/textile/" rel="nofollow" target="_blank">Textile</a> or <a href="http://daringfireball.net/projects/markdown/" rel="nofollow" target="_blank">Markdown</a>. <br /><br />Let’s make a very simple presentation as a proof of concept: we’re going to create a “Python flow control” presentation, showing some basic structures of the language: if, for and while. We need a cover, a slide for each structure (with some topics and code examples) and the last slide for questions and answers. Here is the RST code for it:<a name="more"></a><pre><br />Python<br />======<br /><br />--------------<br /><br />If<br />==<br /><br />* Please don't use ()<br />* Never forget the ``:`` at the end of the line<br /><br />Check this code:<br /><br />.. sourcecode:: python<br /><br /> x, y = 1, 2<br /> if x > y:<br /> print 'x is greater'<br /><br />--------------<br /><br />For<br />===<br /><br />* ``for`` iterates over a sequence<br />* Never forget the ``:`` at the end of the line<br /><br />Check this code:<br /><br />.. sourcecode:: python<br /><br /> numbers = [1, 2, 3, 4, 5,]<br /> for number in numbers:<br /> print number<br /><br />--------------<br /><br />While<br />=====<br /><br />* ``while`` is like ``if``, but executes while the codition is ``True``<br />* please don't use ()<br />* never forget the ``:`` at the end of the line<br /><br />Check this code:<br /><br />.. sourcecode:: python<br /><br /> from random import randint<br /><br /> args = (1, 10,)<br /> x = randint(*args)<br /> while x != 6:<br /> x = randint(*args)<br /><br />--------------<br /><br />Thank you!<br />==========<br /></pre>As you can see it’s very simple. If you’re familiar with RST syntax, you can guess what landslide does: it converts the entire content to HTML and then split it by <code><hr /></code> tag. Each slide will contain two sections: a header and a body. The header contains only an <code><h1></h1></code> element and the body contains everything. <br /><br />We can generate the HTML output by calling the landslide command in the terminal: <pre>% landslide python.rst</pre>To use <code>landslide</code> command, you need to install it. I suggest you do this via pip: <pre>% [sudo] pip install landslide</pre>landslide supports theming, so you can customize it by creating your own theme. Your theme should contain two CSS files: screen.css (for the HTML version of slides) and print.css (for the PDF version of the slides). You might also customize the HTML (base.html) and JS files (slides.js), but you <strong>have</strong> to customize the CSS files in your theme. You specify the theme using the <code>--theme</code> directive. You might want to check all options available in the command line utility using <code>--help</code>: <pre>% landslide --help</pre>It’s quite easy to extend landslide changing its theme or adding new macros. Check the <a href="https://github.com/adamzap/landslide" rel="nofollow" target="_blank">official repository at Github</a>. This example, and a <a href="https://github.com/fsouza/landslide-example/blob/master/python.markdown" rel="nofollow" target="_blank">markdown</a> version for the same example are available in a <a href="https://github.com/fsouza/landslide-example" rel="nofollow" target="_blank">repository</a> in my github profile. <br /><br />You can also see the <a href="http://p.souza.cc/landslide-example/" rel="nofollow" target="_blank">slides live</a>!fsouzanoreply@blogger.comhttp://f.souza.cc/Splinter sprint on FISLtag:blogger.com,1999:blog-4820627057316560708.post-55589841372927218542019-12-10T03:42:20+00:00<img align="left" border="0" height="199" src="http://2.bp.blogspot.com/-CW51FwLjWWU/T_5AM1E-OOI/AAAAAAAAA7M/9-5xr3cLwss/s400/fisl-logo1.jpg" width="200" />We are going to start tomorrow, on <a href="http://fisl.softwarelivre.org/" rel="nofollow" target="_blank">FISL</a>, another splinter sprint. <em>“From June 29 through July 2, 2011, fisl12 will be hosted at the PUC Events Center, in Porto Alegre, Rio Grande do Sul, Brazil”</em> (copied from FISL website). But don’t worry about the location: anyone in anywhere can join us in this sprint. There is an <a href="https://github.com/cobrateam/splinter/wiki/sprint29jun2011">entry</a> in splinter wiki about this sprint, and I’m just replicating the information here...<a name="more"></a><br /><br /><br /><h4>What is a splinter sprint?</h4>Basically, a splinter sprint is an excuse for people to focus their undivided attention, for a set time frame, on improving splinter. It’s a focused, scheduled effort to fix bugs, add new features and improve documentation. <br /><br />Anybody, anywhere around the world, can participate and contribute. If you’ve never contributed to splinter before, this is the perfect chance for you to chip in. <h4>How to contribute</h4><ol> <li>Choose an <a href="https://github.com/cobrateam/splinter/issues">issue</a></li> <li>Create a fork</li> <li>Send a pull request</li></ol><em>Remember: all new features should be well tested and documented. An issue can’t be closed if there isn’t docs for the solution code.</em><h4>Preparing for the sprint</h4>Get an <a href="http://en.wikipedia.org/wiki/Internet_Relay_Chat">IRC</a> client, so that you can join us in the channel <a href="http://cobrateam.info/">#cobrateam</a> on Freenode. <br /><br />See all you there!fsouzanoreply@blogger.comhttp://f.souza.cc/Testing jQuery plugins with Jasminetag:blogger.com,1999:blog-4820627057316560708.post-62133493753578925822019-12-10T03:42:20+00:00<img align="right" border="0" height="48" src="http://2.bp.blogspot.com/-_CNpJ1vbSZs/T_481qLXhnI/AAAAAAAAA68/SnK7hjJRP3g/s400/jasmine_logo.png" width="150" />Since I started working at <a href="http://globo.com" target="_blank">Globo.com</a>, I developed some jQuery plugins (for internal use) with my team, and we are starting to test these plugins using <a href="http://pivotal.github.com/jasmine/" target="_blank">Jasmine</a>, <em>“a behavior-driven development framework for testing your JavaScript code”</em>. In this post, I will show how to develop a very simple jQuery plugin (based on an example that I learned with <a href="http://rdworth.org/" rel="nofollow" target="_blank">Ricard D. Worth</a>): zebrafy. This plugin “zebrafies” a table, applying different classes to odd and even lines. Let’s start setting up a Jasmine environment...<a name="more"></a><br /><br />First step is download the <a href="http://pivotal.github.com/jasmine/download.html" rel="nofollow" target="_blank">standalone version of Jasmine</a>, then extract it and edit the runner. The runner is a simple HTML file, that loads Jasmine and all JavaScript files you want to test. But, wait... why not test using node.js or something like this? Do I really need the browser on this test? You don’t <strong>need</strong>, but I think it is important to test a plugin that works with the DOM using a real browser. Let’s delete some files and lines from <em>SpecRunner.html</em> file, so we adapt it for our plugin. This is how the structure is going to look like: <pre><br />.<br />├── SpecRunner.html<br />├── lib<br />│ ├── jasmine-1.0.2<br />│ │ ├── MIT.LICENSE<br />│ │ ├── jasmine-html.js<br />│ │ ├── jasmine.css<br />│ │ └── jasmine.js<br />│ └── jquery-1.6.1.min.js<br />├── spec<br />│ └── ZebrafySpec.js<br />└── src<br /> └── jquery.zebrafy.js<br /></pre>You can create the files <em>jquery.zebrafy.js</em> and <em>ZebrafySpec.js</em>, but remember: it is <a href="http://f.souza.cc/search/label/bdd">BDD</a>, we need to describe the behavior first, then write the code. So let’s start writing the specs in <em>ZebrafySpec.js</em> file using Jasmine. If you are familiar with <a href="http://rspec.info" rel="nofollow" target="_blank">RSpec</a> syntax, it’s easy to understand how to write spec withs Jasmine, if you aren’t, here is the clue: Jasmine is a lib with some functions used for writing tests in an easier way. I’m going to explain each function “on demmand”, when we need something, we learn how to use it! ;) <br /><br />First of all, we need to start a new test suite. Jasmine provides the <code>describe</code> function for that, this function receives a string and another function (a callback). The string describes the test suite and the function is a callback that delimites the scope of the test suite. Here is the <code>Zebrafy</code> suite: <pre><br />describe('Zebrafy', function () {<br /><br />});<br /></pre>Let’s start describing the behavior we want to get from the plugin. The most basic is: we want different CSS classes for odd an even lines in a table. Jasmine provides the <code>it</code> function for writing the tests. It also receives a string and a callback: the string is a description for the test and the callback is the function executed as test. Here is the very first test: <pre><br />it('should apply classes zebrafy-odd and zebrafy-even to each other table lines', function () {<br /> var table = $("#zebra-table");<br /> table.zebrafy();<br /> expect(table).toBeZebrafyied();<br />});<br /></pre>Okay, here we go: in the first line of the callback, we are using jQuery to select a table using the <code>#zebra-table</code> selector, which will look up for a table with the ID attribute equals to <em>“zebra-table”</em>, but we don’t have this table in the DOM. What about add a new table to the DOM in a hook executed before the test run and remove the table in another hook that runs after the test? Jasmine provide two functions: <code>beforeEach</code> and <code>afterEach</code>. Both functions receive a callback function to be executed and, as the names suggest, the <code>beforeEach</code> callback is called before each test run, and the <code>afterEach</code> callback is called after the test run. Here are the hooks: <pre><br />beforeEach(function () {<br /> $('<table id="zebra-table"></table>').appendTo('body');<br /> for (var i=0; i < 10; i++) {<br /> $('<tr></tr>').append('<td></td>').append('<td></td>').append('<td></td>').appendTo('#zebra-table');<br /> };<br />});<br /><br />afterEach(function () {<br /> $("#zebra-table").remove();<br />});<br /></pre>The <code>beforeEach</code> callback uses jQuery to create a table with 10 rows and 3 columns and add it to the DOM. In <code>afterEach</code> callback, we just remove that table using jQuery again. Okay, now the table exists, let’s go back to the test: <pre><br />it('should apply classes zebrafy-odd and zebrafy-even to each other table lines', function () {<br /> var table = $("#zebra-table");<br /> table.zebrafy();<br /> expect(table).toBeZebrafyied();<br />});<br /></pre>In the second line, we call our plugin, that is not ready yet, so let’s forward to the next line, where we used the <code>expect</code> function. Jasmine provides this function, that receives an object and executes a <em>matcher</em> against it, there is a lot of built-in matchers on Jasmine, but <code>toBeZebrafyied</code> is not a built-in matcher. Here is where we know another Jasmine feature: the capability to write custom matchers, but how to do this? You can call the <code>beforeEach</code> again, and use the <code>addMatcher</code> method of Jasmine object: <pre><br />beforeEach(function () {<br /> this.addMatchers({<br /> toBeZebrafyied: function() {<br /> var isZebrafyied = true;<br /><br /> this.actual.find("tr:even").each(function (index, tr) {<br /> isZebrafyied = $(tr).hasClass('zebrafy-odd') === false && $(tr).hasClass('zebrafy-even');<br /> if (!isZebrafyied) {<br /> return;<br /> };<br /> });<br /><br /> this.actual.find("tr:odd").each(function (index, tr) {<br /> isZebrafyied = $(tr).hasClass('zebrafy-odd') && $(tr).hasClass('zebrafy-even') === false;<br /> if (!isZebrafyied) {<br /> return;<br /> };<br /> });<br /><br /> return isZebrafyied;<br /> }<br /> });<br />});<br /></pre>The method <code>addMatchers</code> receives an object where each property is a matcher. Your matcher can receive arguments if you want. The object being matched can be accessed using <code>this.actual</code>, so here is what the method above does: it takes all odd <code><tr></code> elements of the table (<code>this.actual</code>) and check if them have the CSS class <code>zebrafy-odd</code> and don’t have the CSS class <code>zebrafy-even</code>, then do the same checking with even <code><tr></code> lines. <br /><br />Now that we have wrote the test, it’s time to write the plugin. Here some jQuery code: <pre><br />(function ($) {<br /> $.fn.zebrafy = function () {<br /> this.find("tr:even").addClass("zebrafy-even");<br /> this.find("tr:odd").addClass("zebrafy-odd");<br /> };<br />})(jQuery);<br /></pre>I’m not going to explain <a href="http://docs.jquery.com/Plugins/Authoring" rel="nofollow" target="_blank">how to implement a jQuery plugin</a> neither <a href="http://benalman.com/news/2010/11/immediately-invoked-function-expression/" rel="nofollow" target="_blank">what are those brackets on function</a>, this post aims to show how to use Jasmine to test jQuery plugins. <br /><br />By convention, jQuery plugins are “chainable”, so let’s make sure the zebrafy plugin is chainable using a spec: <pre><br />it('zebrafy should be chainable', function() {<br /> var table = $("#zebra-table");<br /> table.zebrafy().addClass('black-bg');<br /> expect(table.hasClass('black-bg')).toBeTruthy();<br />});<br /></pre>As you can see, we used the built-in matcher <code>toBeTruthy</code>, which asserts that an object or expression is <code>true</code>. All we need to do is return the jQuery object in the plugin and the test will pass: <pre><br />(function ($) {<br /> $.fn.zebrafy = function () {<br /> return this.each(function (index, table) {<br /> $(table).find("tr:even").addClass("zebrafy-even");<br /> $(table).find("tr:odd").addClass("zebrafy-odd");<br /> });<br /> };<br />})(jQuery);<br /></pre>So, the plugin is tested and ready to release! :) You can check the entire code and test with more spec in a <a href="https://github.com/fsouza/jquery-testing-jasmine" target="_blank">Github repository</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Splinter: Python tool for acceptance tests on web applicationstag:blogger.com,1999:blog-4820627057316560708.post-82076759591043623482019-12-10T03:42:20+00:00<img align="right" border="0" src="http://4.bp.blogspot.com/-iNVYTZpX78Y/T_40K_KoEYI/AAAAAAAAA6s/KJzqpmaUNIU/s1600/splinter1.jpg" /><a href="https://github.com/jnicklas/capybara" rel="nofollow" target="_blank">Capybara</a> and <a href="https://github.com/brynary/webrat" rel="nofollow" target="_blank">Webrat</a> are great Ruby tools for acceptance tests. A few months ago, we started a great tool for acceptance tests in <a href="http://python.org/" rel="nofollow" target="_blank">Python</a> web applications, called <a href="http://splinter.cobrateam.info/">Splinter</a>. There are many acceptance test tools on Python world: <a href="http://seleniumhq.org/" rel="nofollow" target="_blank">Selenium</a>, <a href="https://github.com/idealistdev/alfajor" rel="nofollow" target="_blank">Alfajor</a>, <a href="http://www.getwindmill.com/" rel="nofollow" target="_blank">Windmill</a>, <a href="http://wwwsearch.sourceforge.net/mechanize/" rel="nofollow" target="_blank">Mechanize</a>, <a href="http://pypi.python.org/pypi/zope.testbrowser" rel="nofollow" target="_blank">zope.testbrowser</a>, etc. Splinter was not created to be another acceptance tool, but an abstract layer over other tools, its goal is provide a unique API that make acceptance testing easier and funnier.<br /><a name="more"></a><br />In this post, I will show some basic usage of Splinter for simple web application tests. <a href="https://github.com/cobrateam/splinter" rel="nofollow" target="_blank">Splinter</a> is a tool useful on tests of any web application. You can even test a Java web application using Splinter. This post example is a "test" of a Facebook feature, just because I want to focus on how to use Splinter, not on how to write a web application. The feature to be tested is the creation of an event (the Splinter sprint), following all the flow: first the user will login on Facebook, then click on "Events" menu item, then click on "Create an Event" button, enter all event informations and click on "Create event" button. So, let’s do it… <br /><br />First step is create a <a href="http://splinter.cobrateam.info/docs/api/driver-and-element-api.html#splinter.browser.Browser" rel="nofollow" target="_blank">Browser</a> instance, which will provide method for interactions with browser (where the browser is: Firefox, Chrome, etc.). The code we need for it is very simple: <br /><pre>browser = Browser("firefox")</pre><code>Browser</code> is a class and its constructor receives the driver to be used with that instance. Nowadays, there are three drivers for Splinter: firefox, chrome and zope.testbrowser. We are using Firefox, and you can easily use Chrome by simply changing the driver from <code>firefox</code> to <code>chrome</code>. It’s also very simple to add another driver to Splinter, and I plan to cover how to do that in another blog post here. <br /><br />A new browser session is started when we got the <code>browser</code> object, and this is the object used for Firefox interactions. Let's start a new event on Facebook, the Splinter Sprint. First of all, we need to <em>visit</em> the Facebook homepage. There is a <code>visit</code> method on Browser class, so we can use it: <br /><pre>browser.visit("https://www.facebook.com")</pre><code>visit</code> is a blocking operation: it waits for page to load, then we can navigate, click on links, fill forms, etc. Now we have Facebook homepage opened on browser, and you probably know that we need to login on Facebook page, but what if we are already logged in? So, let's create a method that login on Facebook with provided authentication data only the user is not logged in (imagine we are on a TestCase class): <br /><pre>def do_login_if_need(self, username, password):<br /> if self.browser.is_element_present_by_css('div.menu_login_container'):<br /> self.browser.fill('email', username)<br /> self.browser.fill('pass', password)<br /> self.browser.find_by_css('div.menu_login_container input[type="submit"]').first.click()<br /> assert self.browser.is_element_present_by_css('li#navAccount')<br /></pre>What was made here? First of all, the method checks if there is an element present on the page, using a CSS selector. It checks for a <code>div</code> that contains the <em>username</em> and <em>password</em> fields. If that div is present, we tell the browser object to fill those fields, then find the <code>submit</code> button and click on it. The last line is an assert to guarantee that the login was successful and the current page is the Facebook homepage (by checking the presence of “Account” <code>li</code>). <br /><br />We could also <a href="http://splinter.cobrateam.info/docs/finding.html" rel="nofollow" target="_blank">find elements</a> by its texts, labels or whatever appears on screen, but remember: Facebook is an internationalized web application, and we can’t test it using only a specific language. <br /><br />Okay, now we know how to visit a webpage, check if an element is present, fill a form and click on a button. We're also logged in on Facebook and can finally go ahead create the <em>Splinter sprint</em> event. So, here is the event creation flow, for a user: <br /><ol><li>On Facebook homepage, click on “Events” link, of left menu</li><li>The “Events” page will load, so click on “Create an Event” button</li><li>The user see a page with a form to create an event</li><li>Fill the date and chose the time</li><li>Define what is the name of the event, where it will happen and write a short description for it</li><li>Invite some guests</li><li>Upload a picture for the event</li><li>Click on “Create Event” button</li></ol>We are going to do all these steps, except the 6th, because the Splinter Sprint will just be a public event and we don’t need to invite anybody. There are some boring AJAX requests on Facebook that we need to deal, so there is not only Splinter code for those steps above. First step is click on “Events” link. All we need to do is <code>find</code> the link and <code>click</code> on it: <br /><pre>browser.find_by_css('li#navItem_events a').first.click()</pre>The <code>find_by_css</code> method takes a CSS selector and returns an <a href="http://splinter.cobrateam.info/docs/api/element-list.html#splinter.element_list.ElementList" rel="nofollow" target="_blank">ElementList</a>. So, we get the first element of the list (even when the selector returns only an element, the return type is still a <em>list</em>) and click on it. Like <code>visit</code> method, <code>click</code> is a blocking operation: the driver will only listen for new actions when the request is finished (the page is loaded). <br /><br />We’re finally on "new event" page, and there is a form on screen waiting for data of the <em>Splinter Sprint</em>. Let’s fill the form. Here is the code for it: <br /><pre>browser.fill('event_startIntlDisplay', '5/21/2011')<br />browser.select('start_time_min', '480')<br />browser.fill('name', 'Splinter sprint')<br />browser.fill('location', 'Rio de Janeiro, Brazil')<br />browser.fill('desc', 'For more info, check out the #cobratem channel on freenode!')<br /></pre>That is it: the event is going to happen on May 21th 2011, at 8:00 in the morning (480 minutes). As we know, the event name is <em>Splinter sprint</em>, and we are going to join some guys down here in Brazil. We filled out the form using <code>fill</code> and <code>select</code> methods. <br /><br />The <code>fill</code> method is used to fill a "fillable" field (a textarea, an input, etc.). It receives two strings: the first is the <em>name</em> of the field to fill and the second is the <em>value</em> that will fill the field. <code>select</code> is used to select an option in a select element (a “combo box”). It also receives two string parameters: the first is the <em>name</em> of the select element, and the second is the <em>value</em> of the option being selected. <br /><br />Imagine you have the following select element: <br /><pre><select name="gender"><br /> <option value="m">Male</option><br /> <option value="f">Female</option><br /></select><br /></pre>To select “Male”, you would call the select method this way: <br /><pre>browser.select("gender", "m")</pre>The last action before click on “Create Event” button is upload a picture for the event. On new event page, Facebook loads the file field for picture uploading inside an <code>iframe</code>, so we need to switch to this frame and interact with the form present inside the frame. To show the frame, we need to click on “Add Event Photo” button and then switch to it, we already know how click on a link: <br /><pre>browser.find_by_css('div.eventEditUpload a.uiButton').first.click()</pre>When we click this link, Facebook makes an asynchronous request, which means the driver does not stay blocked waiting the end of the request, so if we try to interact with the frame BEFORE it appears, we will get an <code>ElementDoesNotExist</code> exception. Splinter provides the <code>is_element_present</code> method that receives an argument called <code>wait_time</code>, which is the time Splinter will wait for the element to appear on the screen. If the element does not appear on screen, we can’t go on, so we can assume the test failed (remember we are testing a Facebook feature): <br /><pre>if not browser.is_element_present_by_css('iframe#upload_pic_frame', wait_time=10):<br />fail("The upload pic iframe did'n't appear :(")<br /></pre>The <code>is_element_present_by_css</code> method takes a CSS selector and tries to find an element using it. It also receives a <code>wait_time</code> parameter that indicates a time out for the search of the element. So, if the <code>iframe</code> element with <em>ID=”upload_pic_frame”</em> is not present or doesn’t appear in the screen after 10 seconds, the method returns <code>False</code>, otherwise it returns <code>True</code>. <br /><blockquote><strong>Important:</strong> <code>fail</code> is a pseudocode sample and doesn’t exist (if you’re using <code>unittest</code> library, you can invoke <code>self.fail</code> in a TestCase, exactly what I did in <a href="https://github.com/cobrateam/splinter/blob/master/samples/test_facebook_events.py" rel="nofollow" target="_blank" title="Snippet for creating a new event on Facebook using Splinter">complete snippet for this example</a>, available at Github).</blockquote>Now we see the <code>iframe</code> element on screen and we can finally upload the picture. Imagine we have a variable that contains the path of the picture (and not a file object, <code>StringIO</code>, or something like this), and this variable name is <code>picture_path</code>, this is the code we need: <br /><pre>with browser.get_iframe('upload_pic_frame') as frame:<br /> frame.attach_file('pic', picture_path)<br /> time.sleep(10)<br /></pre>Splinter provides the <code>get_iframe</code> method that changes the context and returns another objet to interact with the content of the frame. So we call the <code>attach_file</code> method, who also receives two strings: the first is the <em>name</em> of the input element and the second is the absolute <em>path</em> to the file being sent. Facebook also uploads the picture asynchronously, but there’s no way to wait some element to appear on screen, so I just put Python to sleep 10 seconds on last line. <br /><br />After finish all these steps, we can finally click on “Create Event” button and asserts that Facebook created it: <br /><pre>browser.find_by_css('label.uiButton input[type="submit"]').first.click()<br />title = browser.find_by_css('h1 span').first.text<br />assert title == 'Splinter sprint'<br /></pre>After create an event, Facebook redirects the browser to the event page, so we can check if it really happened by asserting the header of the page. That’s what the code above does: in the new event page, it click on submit button, and after the redirect, get the text of a span element and asserts that this text equals to <em>“Splinter sprint”</em>. <br /><br />That is it! This post was an overview on Splinter API. Check out the <a href="https://github.com/cobrateam/splinter/blob/master/samples/test_facebook_events.py" target="_blank">complete snippet</a>, written as a test case and also check out <a href="https://github.com/cobrateam/splinter" target="_blank">Splinter repository at Github</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Killer Java applications server with nginx and memcachedtag:blogger.com,1999:blog-4820627057316560708.post-62549373946802812872019-12-10T03:42:20+00:00<span>Last days I worked setting up a new web serving structure for </span><a href="http://www.wine.com.br/" rel="nofollow" target="_blank">Wine</a><span>, the largest wine’s e-commerce in Latin America. After testing, studying and learning a lot, we built a nice solution based on </span><a href="http://nginx.org/" rel="nofollow" target="_blank">nginx</a><span> and </span><a href="http://memcached.org/" rel="nofollow" target="_blank">memcached</a><span>. I will use a picture to describe the architecture:</span><br /><br /><div class="separator"><img border="0" src="http://3.bp.blogspot.com/-5suvrwrJD4A/T_4f_sK28aI/AAAAAAAAA6c/Ok4B3pr5mZ8/s1600/nginx-tomcat-memcached-e1292107045545.png" /></div><span>As you can see, when a client do a request to the nginx server, it first checks on memcached if the response is already cached. If the response was not found on cache server, then nginx forward the request to <a href="http://tomcat.apache.org/" rel="nofollow" target="_blank">Tomcat</a>, which process the request, cache the response on memcached and returns it to nginx. Tomcat works only for the first client, and all other clients requesting the same resource will get the cached response on RAM. My objective with this post is to show how we built this architecture.</span><a name="more"></a><h3>nginx</h3>nginx was compiled following <a href="http://library.linode.com/web-servers/nginx/installation/ubuntu-9.10-karmic#installing_nginx_from_the_source_distribution" rel="nofollow" target="_blank">Linode instructions for nginx installation from source</a>. The only difference is that we added the <a href="http://wiki.nginx.org/HttpMemcachedModule" rel="nofollow" target="_blank">nginx memcached module</a>. So, first I downloaded the <code>memc_module</code> source from Github and then built nginx with it. Here is the commands for compiling nginx with memcached module: <br /><pre>% ./configure --prefix=/opt/nginx --user=nginx --group=nginx --with-http_ssl_module --add-module={your memc_module source path}<br />% make<br />% sudo make install<br /></pre>After install nginx and <a href="http://library.linode.com/web-servers/nginx/installation/ubuntu-9.10-karmic#create_an_init_script_to_manage_nginx" rel="nofollow" target="_blank">create an init script for it</a>, we can work on its settings for integration with Tomcat. Just for working with separate settings, we changed the <em>nginx.conf</em> file (located in <em>/opt/nginx/conf</em> directory), and it now looks like this: <br /><pre>user nginx;<br />worker_processes 1;<br /><br />error_log logs/error.log;<br /><br />events {<br /> worker_connections 1024;<br />}<br /><br />http {<br /> include mime.types;<br /> default_type application/octet-stream;<br /><br /> log_format main '$remote_addr - $remote_user [$time_local] "$request" '<br /> '$status $body_bytes_sent "$http_referer" '<br /> '"$http_user_agent" "$http_x_forwarded_for"';<br /><br /> access_log logs/access.log main;<br /><br /> sendfile on;<br /> #tcp_nopush on;<br /><br /> #keepalive_timeout 0;<br /> keepalive_timeout 65;<br /><br /> #gzip on;<br /><br /> include /opt/nginx/sites-enabled/*;<br />}<br /></pre>See the last line inside <code>http</code> section: this line tells nginx to include all settings present in the <code>/opt/nginx/sites-enabled</code> directory. So, now, let’s create a default file in this directory, with this content: <br /><pre>server {<br /> listen 80;<br /> server_name localhost;<br /><br /> default_type text/html;<br /><br /> location / {<br /> proxy_set_header X-Real-IP $remote_addr;<br /> proxy_set_header Host $http_host;<br /> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br /><br /> if ($request_method = POST) {<br /> proxy_pass http://localhost:8080;<br /> break;<br /> }<br /><br /> set $memcached_key "$uri";<br /> memcached_pass 127.0.0.1:11211;<br /><br /> error_page 501 404 502 = /fallback$uri;<br /> }<br /><br /> location /fallback/ {<br /> internal; <br /><br /> proxy_set_header X-Real-IP $remote_addr;<br /> proxy_set_header Host $http_host;<br /> proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br /> proxy_redirect off;<br /><br /> proxy_pass http://localhost:8080;<br /> }<br /><br />}<br /></pre>Some stuffs must be explained here: the <code>default_type</code> directive is necessary for proper serving of cached responses (if you are cache other content types like <code>application/json</code> or <code>application/xml</code>, you should take a look at nginx documentation and deal conditionally with content types). The <code>location /</code> scope defines some settings for proxy, like IP and host. We just did it because we need to pass the right information to our backend (Tomcat or memcached). See more about <code>proxy_set_header</code> at <a href="http://wiki.nginx.org/HttpProxyModule#proxy_set_header" rel="nofollow" target="_blank">nginx documentation</a>. After that, there is a simple verification oF the request method. We don’t want to cache <em>POST</em> requests. <br /><br />Now we get the magic: first we set the <code>$memcached_key</code> and then we use the <code>memcached_pass</code> directive, the <code>$memcached_key</code> is the URI. <code>memcached_pass</code> is very similar to <code>proxy_pass</code>, nginx “proxies” the request to <code>memcached</code>, so we can get some HTTP status code, like 200, 404 or 502. We define error handlers for two status codes: <ul><li>404: memcached module returns a 404 error when the key is not on memcached server;</li><li>502: memcached module returns a 502 error when it can’t found memcached server.</li></ul>So, when nginx gets any of those errors, it should forward the request to Tomcat, creating another proxy. We configured it out on <code>fallback</code>, an <a href="http://wiki.nginx.org/HttpCoreModule#internal" rel="nofollow" target="_blank">internal location</a> that builds a proxy between nginx and Tomcat (listening on port 8080). Everything is set up with nginx. As you can see in the picture or in the nginx configuration file, nginx doesn’t write anything to memcached, it only reads from memcached. The application should write to memcached. Let’s do it. <h3>Java application</h3>Now is the time to write some code. I chose an application written by a friend of mine. It’s a very simple CRUD of users, built by <a href="http://wbotelhos.com/" rel="nofollow" target="_blank">Washington Botelho</a> with the goal of introducing <a href="http://vraptor.org/" rel="nofollow" target="_blank">VRaptor</a>, a powerful and fast development focused web framework. Washington also wrote a blog post explaining the application, if you don’t know VRaptor or want to know how the application was built, check the blog post <a href="http://www.wbotelhos.com/2010/11/24/getting-started-with-vraptor-3/" rel="nofollow" target="_blank">"Getting started with VRaptor 3"</a>. I <a href="https://github.com/fsouza/starting-with-vraptor-3" rel="nofollow" target="_blank">forked</a> the application, made some minor changes and added a magic filter for caching. All Java code that I want to show here is the filter code: <pre><br />package com.franciscosouza.memcached.filter;<br /><br />import java.io.IOException;<br />import java.io.PrintWriter;<br />import java.io.StringWriter;<br />import java.net.InetSocketAddress;<br /><br />import javax.servlet.Filter;<br />import javax.servlet.FilterChain;<br />import javax.servlet.FilterConfig;<br />import javax.servlet.ServletException;<br />import javax.servlet.ServletOutputStream;<br />import javax.servlet.ServletRequest;<br />import javax.servlet.ServletResponse;<br />import javax.servlet.http.HttpServletRequest;<br />import javax.servlet.http.HttpServletResponse;<br />import javax.servlet.http.HttpServletResponseWrapper;<br /><br />import net.spy.memcached.MemcachedClient;<br /><br />/**<br /> * Servlet Filter implementation class MemcachedFilter<br /> */<br />public class MemcachedFilter implements Filter {<br /><br /> private MemcachedClient mmc;<br /><br /> static class MemcachedHttpServletResponseWrapper extends HttpServletResponseWrapper {<br /><br /> private StringWriter sw = new StringWriter();<br /><br /> public MemcachedHttpServletResponseWrapper(HttpServletResponse response) {<br /> super(response);<br /> }<br /><br /> public PrintWriter getWriter() throws IOException {<br /> return new PrintWriter(sw);<br /> }<br /><br /> public ServletOutputStream getOutputStream() throws IOException {<br /> throw new UnsupportedOperationException();<br /> }<br /><br /> public String toString() {<br /> return sw.toString();<br /> }<br /> }<br /><br /> /**<br /> * Default constructor.<br /> */<br /> public MemcachedFilter() {<br /> }<br /><br /> /**<br /> * @see Filter#destroy()<br /> */<br /> public void destroy() {<br /> }<br /><br /> /**<br /> * @see Filter#doFilter(ServletRequest, ServletResponse, FilterChain)<br /> */<br /> public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException {<br /> MemcachedHttpServletResponseWrapper wrapper = new MemcachedHttpServletResponseWrapper((HttpServletResponse) response);<br /> chain.doFilter(request, wrapper);<br /><br /> HttpServletRequest inRequest = (HttpServletRequest) request;<br /> HttpServletResponse inResponse = (HttpServletResponse) response;<br /><br /> String content = wrapper.toString();<br /><br /> PrintWriter out = inResponse.getWriter();<br /> out.print(content);<br /><br /> if (!inRequest.getMethod().equals("POST")) {<br /> String key = inRequest.getRequestURI();<br /> mmc.set(key, 5, content);<br /> }<br /> }<br /><br /> /**<br /> * @see Filter#init(FilterConfig)<br /> */<br /> public void init(FilterConfig fConfig) throws ServletException {<br /> try {<br /> mmc = new MemcachedClient(new InetSocketAddress("localhost", 11211));<br /> } catch (IOException e) {<br /> e.printStackTrace();<br /> throw new ServletException(e);<br /> }<br /> }<br />}<br /></pre>First, the dependency: for memcached communication, we used <a href="http://code.google.com/p/spymemcached/" rel="nofollow" target="_blank">spymemcached</a> client. It is a simple and easy to use memcached library. I won’t explain all the code, line by line, but I can tell the idea behind the code: first, call <code>doFilter</code> method on <code>FilterChain</code>, because we want to get the response and work with that. Take a look at the <code>MemcachedHttpServletResponseWrapper</code> instance, it encapsulates the response and makes easier to play with response content. <br /><br />We get the content, write it on response writer and put it in cache using the <code>MemcachedClient</code> provided by <em>spymemcached</em>. The request URI is the key and timeout is 5 seconds. <h3>web.xml</h3>Last step is to add the filter on <em>web.xml</em> file of the project, map it before the VRaptor filter is very important for proper working: <pre><br /><?xml version="1.0" encoding="UTF-8"?><br /><web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5"><br /> <display-name>memcached sample</display-name><br /><br /> <filter><br /> <filter-name>vraptor</filter-name><br /> <filter-class>br.com.caelum.vraptor.VRaptor</filter-class><br /> </filter><br /> <br /> <b><filter><br /> <filter-name>memcached</filter-name><br /> <filter-class>com.franciscosouza.memcached.filter.MemcachedFilter</filter-class><br /> </filter><br /> <br /> <filter-mapping><br /> <filter-name>memcached</filter-name><br /> <url-pattern>/*</url-pattern><br /> </filter-mapping></b><br /><br /> <filter-mapping><br /> <filter-name>vraptor</filter-name><br /> <url-pattern>/*</url-pattern><br /> <dispatcher>FORWARD</dispatcher><br /> <dispatcher>REQUEST</dispatcher><br /> </filter-mapping><br /><br /></web-app><br /></pre>That is it! Now you can just run Tomcat on port <code>8080</code> and nginx on port <code>80</code>, and access <code>http://localhost</code> on your browser. Try some it: raise up the cache timeout, navigate on application and turn off Tomcat. You will still be able to navigate on some pages that use GET request method (users list, home and users form). <br /><br />Check the entire code out on Github: <a href="https://github.com/fsouza/starting-with-vraptor-3" rel="nofollow" target="_blank">https://github.com/fsouza/starting-with-vraptor-3</a>. If you have any questions, troubles or comments, please let me know! ;)fsouzanoreply@blogger.comhttp://f.souza.cc/Flying with tipfy on Google App Enginetag:blogger.com,1999:blog-4820627057316560708.post-57151148213531150802019-12-10T03:42:20+00:00<div class="separator"><a href="http://1.bp.blogspot.com/-HbEeNEBkX3k/T_4LzIjjtYI/AAAAAAAAA6Q/cS-brE8Sjes/s1600/tipfy_logo.png"><img border="0" src="http://1.bp.blogspot.com/-HbEeNEBkX3k/T_4LzIjjtYI/AAAAAAAAA6Q/cS-brE8Sjes/s1600/tipfy_logo.png" /></a></div><span id="goog_1563938629"></span>Hooray, there is a bonus part in the series (after a looooooooooooong wait)! In the first blog post, about Django, I received a comment about the use of <a href="http://www.tipfy.org/" rel="nofollow" target="_blank">tipfy</a>, a small Python web framework made specifically for Google App Engine. Like <a href="http://f.souza.cc/2010/08/flying-with-flask-on-google-app-engine.html">Flask</a>, tipfy is not a full stack framework and we will not use a database abstraction layer, we will use just the <a href="https://developers.google.com/appengine/docs/python/datastore/" rel="nofollow" target="_blank">Google App Engine Datastore API</a>, but tipfy was designed for Google App Engine, so it is less laborious to work with tipfy on App Engine.<br /><a name="more"></a><br />First, we have to download tipfy. There are two options on official tipfy page: an all-in-one package and a do-it-yourself packaged. I am lazy, so I downloaded and used the all-in-one package. That is so easy: <br /><pre>% wget http://www.tipfy.org/tipfy.build.tar.gz<br />% tar -xvzf tipfy.0.6.2.build.tar.gz<br />% mv project gaeseries<br /></pre>After it, we go to the project folder and see the <a href="http://www.tipfy.org/wiki/guide/project-structure/" rel="nofollow" target="_blank">project structure</a> provided by tipfy. There is a directory called <em>"app"</em>, where the App Engine app is located. The <em>app.yaml</em> file is in the <em>app</em> directory, so we open that file and change the application id and the application version. Here is the <em>app.yaml</em> file: <br /><pre>application: gaeseries<br />version: 4<br />runtime: python<br />api_version: 1<br /><br />derived_file_type:<br />- python_precompiled<br /><br />handlers:<br />- url: /(robots\.txt|favicon\.ico)<br /> static_files: static/\1<br /> upload: static/(.*)<br /><br />- url: /remote_api<br /> script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py<br /> login: admin<br /><br />- url: /_ah/queue/deferred<br /> script: main.py<br /> login: admin<br /><br />- url: /.* <br /> script: main.py<br /></pre>After this, we can start to code our application. tipfy deals with requests using <i>handlers</i>. A handler is a class that has methods to deal with different kinds of requests. That remember me a little the Strut Actions (blergh), but tipfy is a Python framework, what means that it is easier to build web application using it!<br /><br />Understanding tipfy: a URL is mapped to a handler that do something with the request and returns a response. So, we have to create two handlers: one to the list of posts and other to create a post, but let’s create first an application called <i>blog</i>, and a model called <i>Post</i>. Like Django, Flask and web2py, tipfy also works with applications inside a project.<br /><br />To create an application, we just need to create a new Python package with the application name: <br /><pre>% mkdir blog<br />% touch blog/__init__.py<br /></pre>After create the application structure, we install it by putting the application inside the <em>"apps_installed"</em> list on <em>config.py</em> file: <br /><pre># -*- coding: utf-8 -*-<br />"""<br /> config<br /> ~~~~~~<br /><br /> Configuration settings.<br /><br /> :copyright: 2009 by tipfy.org.<br /> :license: BSD, see LICENSE for more details.<br />"""<br />config = {}<br /><br /># Configurations for the 'tipfy' module.<br />config['tipfy'] = {<br /> # Enable debugger. It will be loaded only in development.<br /> 'middleware': [<br /> 'tipfy.ext.debugger.DebuggerMiddleware',<br /> ],<br /> # Enable the Hello, World! app example.<br /> 'apps_installed': [<br /> 'apps.hello_world',<br /> 'apps.blog',<br /> ],<br />}<br /></pre>See the line 22. Inside the application folder, let’s create a Python module called <i>models.py</i>. This module is exactly the same of <a href="http://f.souza.cc/2010/08/flying-with-flask-on-google-app-engine.html">Flask post</a>: <br /><pre>from google.appengine.ext import db<br /><br />class Post(db.Model):<br /> title = db.StringProperty(required = True)<br /> content = db.TextProperty(required = True)<br /> when = db.DateTimeProperty(auto_now_add = True)<br /> author = db.UserProperty(required = True)<br /></pre>After create the model, let’s start building the project by creating the post listing handler. The handlers will be in a module called <i>handlers.py</i>, inside the application folder. Here is the <i>handlers.py</i> code: <br /><pre># -*- coding: utf-8 -*-<br />from tipfy import RequestHandler<br />from tipfy.ext.jinja2 import render_response<br />from models import Post<br /><br />class PostListingHandler(RequestHandler):<br /> def get(self):<br /> posts = Post.all()<br /> return render_response('list_posts.html', posts=posts)<br /></pre>See that we get a list containing all posts from the database and send it to the <i>list_posts.html</i> template. Like Flask, tipfy uses Jinja2 as template engine by default. Following the same way, let’s create a <i>base.html</i> file who represents the layout of the project. This file should be inside the <i>templates</i> folder and contains the following code: <br /><pre><html><br /> <head><br /> <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><br /> <title>{% block title %}{% endblock %}</title><br /> </head><br /> <body id=""><br /> {% block content %}{% endblock %}<br /> </body><br /></html><br /></pre>And now we can create the <i>list_posts.html</i> template extending the base.html template: <br /><pre>{% extends "base.html" %}<br /><br />{% block title %}<br /> Posts list<br />{% endblock %}<br /><br />{% block content %}<br /> Listing all posts:<br /><br /> <ul><br /> {% for post in posts %}<br /> <li><br /> {{ post.title }} (written by {{ post.author.nickname() }})<br /> {{ post.content }}<br /> </li><br /> {% endfor %}<br /> </ul><br />{% endblock %}<br /></pre>Can we access the list of posts now by the URL? No, we can’t yet. Now we have to map the handler to a URL, and we will be able to access the list of posts through the browser. On tipfy, all URL mappings of an application are located in a Python module called <em>urls.py</em>. Create it with the following code: <br /><pre>from tipfy import Rule<br /><br />def get_rules(app):<br /> rules = [<br /> Rule('/posts', endpoint='post-listing', handler='apps.blog.handlers.PostListingHandler'),<br /> ]<br /><br /> return rules<br /></pre>It is very simple: a Python module containing a function called <em>get_rules</em>, that receives the app object as parameter and return a list containing the rules of the application (each rule is an instance of <em>tipfy.Rule</em> class). Now we can finally see the empty post list on the browser, by running the App Engine development server and touching the <em>http://localhost:8080/posts</em> URL on the browser. Run the following command on the project root: <br /><pre>% /usr/local/google_appengine/dev_appserver.py app</pre>And check the browser at <em>http://localhost:8080/posts</em>. And we see the empty list. Now, let’s create the protected handler which will create a new post. tipfy has an <a href="http://www.tipfy.org/wiki/extensions/auth/" rel="nofollow" target="_blank">auth extension</a>, who makes very easy to deal with authentication using the native Google App Engine users API. To use that, we need to configure the session extension, changing the <em>conf.py</em> module, by adding the following code lines: <br /><pre>config['tipfy.ext.session'] = {<br /> 'secret_key' : 'just_dev_testH978DAGV9B9sha_W92S',<br />}<br /></pre>Now we are ready to create the <em>NewPostHandler</em>. We will need to deal with forms, and tipfy has an extension for integration with WTForms, so we have to download and install WTForms and that extension in the project: <br /><pre>% wget http://bitbucket.org/simplecodes/wtforms/get/tip.tar.bz2<br />% tar -xvf tip.tar.bz2<br />% cp -r wtforms/wtforms/ ~/Projetos/gaeseries/app/lib/<br />% wget http://pypi.python.org/packages/source/t/tipfy.ext.wtforms/tipfy.ext.wtforms-0.6.tar.gz<br />% tar -xvzf tipfy.ext.wtforms-0.6.tar.gz<br />% cp -r tipfy.ext.wtforms-0.6/tipfy ~/Projetos/gaeseries/app/distlib<br /></pre>Now we have WTForms extension installed and ready to be used. Let’s create the PostForm class, and then create the handler. I put both classes in the handlers.py file (yeah, including the form). Here is the <em>PostForm</em> class code: <br /><pre>class PostForm(Form):<br /> csrf_protection = True<br /> title = fields.TextField('Title', validators=[validators.Required()])<br /> content = fields.TextAreaField('Content', validators=[validators.Required()])<br /></pre>Add this class to the <em>handlers.py</em> module: <br /><pre>class NewPostHandler(RequestHandler, AppEngineAuthMixin, AllSessionMixins):<br /> middleware = [SessionMiddleware]<br /><br /> @login_required<br /> def get(self, **kwargs):<br /> return render_response('new_post.html', form=self.form)<br /><br /> @login_required<br /> def post(self, **kwargs):<br /> if self.form.validate():<br /> post = Post(<br /> title = self.form.title.data,<br /> content = self.form.content.data,<br /> author = self.auth_session<br /> )<br /> post.put()<br /> return redirect('/posts')<br /> return self.get(**kwargs)<br /><br /> @cached_property<br /> def form(self):<br /> return PostForm(self.request)<br /></pre>A lot of news here: first, tipfy explores the multi-inheritance Python feature and if you will use the auth extension by the native App Engine users API, you have to create you handler class extending <em>AppEngineAuthMixin</em> and <em>AllSessionMixins</em> classes, and add to the <em>middleware</em> list the <em>SessionMiddleware</em> class. See more at the <a href="http://www.tipfy.org/docs/" rel="nofollow" target="_blank">tipfy docs</a>.<br /><br />The last step is create the <i>new_post.html</i> template and deploy the application. Here is the <i>new_post.html</i> template code: <br /><pre>{% extends "base.html" %}<br /><br />{% block title %}<br /> New post<br />{% endblock %}<br /><br />{% block content %}<br /> <form action="" method="post" accept-charset="utf-8"><br /> <p><br /> <label for="title">{{ form.title.label }}</label><br /><br /> {{ form.title|safe }}<br /><br /> {% if form.title.errors %}<br /> <ul class="errors"><br /> {% for error in form.title.errors %}<br /> <li>{{ error }}</li><br /> {% endfor %}<br /> </ul><br /> {% endif %}<br /> </p><br /> <p><br /> <label for="content">{{ form.content.label }}</label><br /><br /> {{ form.content|safe }}<br /><br /> {% if form.content.errors %}<br /> <ul class="errors"><br /> {% for error in form.content.errors %}<br /> <li>{{ error }}</li><br /> {% endfor %}<br /> </ul><br /> {% endif %}<br /> </p><br /> <p><input type="submit" value="Save post"/></p><br /> </form><br />{% endblock %}<br /></pre>Now, we can deploy the application on Google App Engine by simply running this command: <br /><pre>% /usr/local/google_appengine/appcfg.py update app</pre>And you can check the deployed application live here: <a href="http://4.latest.gaeseries.appspot.com/" target="_blank">http://4.latest.gaeseries.appspot.com</a>.<br /><br />The code is available at Github: <a href="https://github.com/fsouza/gaeseries/tree/tipfy">https://github.com/fsouza/gaeseries/tree/tipfy</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Flying with Flask on Google App Enginetag:blogger.com,1999:blog-4820627057316560708.post-82796318493434951822019-12-10T03:42:20+00:00<img align="right" border="0" src="http://2.bp.blogspot.com/-CCvhl4AC2H0/T_3_a80I0HI/AAAAAAAAA6E/un3ca0TcESc/s1600/flask_logo.png" />A little late, finally I introduce the third part of using Python frameworks in Google App Engine. I wrote before about <a href="http://f.souza.cc/2010/08/flying-with-web2py-on-google-app-engine.html">web2py</a> and <a href="http://f.souza.cc/2010/08/flying-with-django-on-google-app-engine.html">Django</a>, and now is the time of <a href="http://flask.pocoo.org/" rel="nofollow" target="_blank">Flask</a>, a Python microframework based on <a href="http://werkzeug.pocoo.org/" rel="nofollow" target="_blank">Werkzeug</a>, <a href="http://jinja.pocoo.org/2/" rel="nofollow" target="_blank">Jinja2</a> and good intentions. Unlike Django and web2py, Flask is not a full stack framework, it has not a database abstraction layer or an object relational mapper, Flask is totally decoupled from model layer. It is really good, because we can use the power of <a href="http://sqlalchemy.org/" rel="nofollow" target="_blank">SQLAlchemy</a> when we are working with relational databases, and when work with non-relational databases, we can use the native API.<br /><a name="more"></a><br />Flask is a microframework, what means that we have more power on customizing the applications, but it is also a little more painful to build an application, because the framework is not a father that does about 10 billion of things for us: it is simple, but still fun! As Flask has no data abstraction layer, we will use the <a href="https://developers.google.com/appengine/docs/python/datastore/" rel="nofollow" target="_blank">BigTable API</a> directly. <br /><br />So, as we have done in other parts of the series, the sample application will be a very simple blog, with a public view listing all posts and other login protected <i>view</i> used for writing posts. The first step is to setup the environment. It is very simple, but I little laborious: first we create an empty directory and put the <i>app.yaml</i> file inside it (yes, we will build everything from scratch). Here is the <i>app.yaml</i> code: <br /><pre>application: gaeseries<br />version: 3<br />runtime: python<br />api_version: 1<br /><br />handlers:<br />- url: .*<br /> script: main.py<br /></pre>We just set the application ID, the version and the URL handlers. We will handle all request in <i>main.py</i> file. Late on this post, I will show the <i>main.py</i> module, the script that handles Flask with Google App Engine. Now, let’s create the Flask application, and deal with App Engine later :)<br /><br />Now we need to install Flask inside the application, so we <a href="http://github.com/mitsuhiko/flask/downloads" rel="nofollow" target="_blank">get Flask from Github</a> (I used 0.6 version), extract it and inside the <i>flask</i> directory get the <i>flask</i> subdirectory. Because <i>Flask</i> depends on <i>Werkzeug</i> and <i>Jinja2</i>, and <i>Jinja2</i> depends on <i>simplejson</i>, you need to get these libraries and install in your application too. Here is how you can get everything: <br /><pre>% wget http://github.com/mitsuhiko/flask/zipball/0.6<br />% unzip mitsuhiko-flask-0.6-0-g5cadd9d.zip<br />% cp -r mitsuhiko-flask-5cadd9d/flask ~/Projetos/blog/gaeseries<br />% wget http://pypi.python.org/packages/source/W/Werkzeug/Werkzeug-0.6.2.tar.gz<br />% tar -xvzf Werkzeug-0.6.2.tar.gz<br />% cp -r Werkzeug-0.6.2/werkzeug ~/Projetos/blog/gaeseries/<br />% wget http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.5.tar.gz<br />% tar -xvzf Jinja2-2.5.tar.gz<br />% cp -r Jinja2-2.5/jinja2 ~/Projetos/blog/gaeseries/<br />% wget http://pypi.python.org/packages/source/s/simplejson/simplejson-2.1.1.tar.gz<br />% tar -xvzf simplejson-2.1.1.tar.gz<br />% cp -r simplejson-2.1.1/simplejson ~/Projetos/blog/gaeseries/<br /></pre>On my computer, the project is under <em>~/Projetos/blog/gaeseries</em>, put all downloaded tools on the root of your application. Now we have everything that we need to start to create our Flask application, so let’s create a Python package called <em>blog</em>, it will be the application directory: <br /><pre>% mkdir blog<br />% touch blog/__init__.py<br /></pre>Inside the <em>__init__.py</em> module, we will create our Flask application and start to code. Here is the <em>__init__.py</em> code: <br /><pre>from flask import Flask<br />import settings<br /><br />app = Flask('blog')<br />app.config.from_object('blog.settings')<br /><br />import views<br /></pre>We imported two modules: <em>settings</em> and <em>views</em>. So we should create the two modules, where we will put the application settings and the views of applications (look that Flask deals in the same way that Django, calling “views” functions that receives a request and returns a response, instead of call it “actions” (like web2py). Just create the files: <br /><pre>% touch blog/views.py<br />% touch blog/settings.py<br /></pre>Here is the <em>settings.py</em> sample code: <br /><pre>DEBUG=True<br />SECRET_KEY='dev_key_h8hfne89vm'<br />CSRF_ENABLED=True<br />CSRF_SESSION_LKEY='dev_key_h8asSNJ9s9=+'<br /></pre>Now is the time to define the model <em>Post</em>. We will define our models inside the application directory, in a module called <em>models.py</em>: <br /><pre>from google.appengine.ext import db<br /><br />class Post(db.Model):<br /> title = db.StringProperty(required = True)<br /> content = db.TextProperty(required = True)<br /> when = db.DateTimeProperty(auto_now_add = True)<br /> author = db.UserProperty(required = True)<br /></pre>The last property is a <em>UserProperty</em>, a “foreign key” to a user. We will use the <a href="https://developers.google.com/appengine/docs/python/users/" rel="nofollow" target="_blank">Google App Engine users API</a>, so the datastore API provides this property to establish a relationship between custom models and the Google account model.<br /><br />We have defined the model, and we can finally start to create the application’s views. Inside the <i>views</i> module, let’s create the public view with all posts, that will be accessed by the URL <i>/posts</i>: <br /><pre>from blog import app<br />from models import Post<br />from flask import render_template<br /><br />@app.route('/posts')<br />def list_posts():<br /> posts = Post.all()<br /> return render_template('list_posts.html', posts=posts)<br /></pre>On the last line of the view, we called the function <em>render_template</em>, which renders a template. The first parameter of this function is the template to be rendered, we passed the <em>list_posts.html</em>, so let’s create it using the Jinja2 syntax, inspired by Django templates. Inside the application directory, create a subdirectory called <em>templates</em> and put inside it a HTML file called <em>base.html</em>. That file will be the application layout and here is its code: <br /><pre><html><br /> <head><br /> <meta http-equiv="Content-type" content="text/html; charset=utf-8"/><br /> <title>{% block title %}Blog{% endblock %}</title><br /> </head><br /> <body><br /> {% block content %}{% endblock %}<br /> </body><br /></html><br /></pre>And now create the <em>list_posts.html</em> template, with the following code: <br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /><ul><br /> {% for post in posts %}<br /> <li><br /> {{ post.title }} (written by {{ post.author.nickname() }})<br /><br /> {{ post.content }}<br /> </li><br /> {% endfor %}<br /></ul><br />{% endblock %}<br /></pre>Now, to test it, we need to run Google App Engine development server on localhost. The <em>app.yaml</em> file defined a <em>main.py</em> script as handler for all requests, so to use Google App Engine local development server, we need to create the <em>main.py</em> file that run our application. Every Flask application is a WSGI application, so we can use an App Engine tool for running WSGI application. In that way, the <em>main.py</em> script is really simple: <br /><pre>from google.appengine.ext.webapp.util import run_wsgi_app<br />from blog import app<br /><br />run_wsgi_app(app)<br /></pre>The script uses the <em>run_wsgi_app</em> function provided by <em>webapp</em>, the built-in Google Python web framework for App Engine. Now, we can run the application in the same way that we ran in the web2py post: <br /><pre>% /usr/local/google_appengine/dev_appserver.py .</pre>And if you access the URL <em>http://localhost:8080/posts</em> in your browser, you will see a blank page, just because there is no posts on the database. Now we will create a login protected view to write and save a post on the database. Google App Engine does not provide a decorator for validate when a user is logged, and Flask doesn’t provide it too. So, let’s create a function decorator called <em>login_required</em> and decorate the view <em>new_post</em> with that decorator. I created the decorator inside a <em>decorators.py</em> module and import it inside the <em>views.py</em> module. Here is the <em>decorators.py</em> code: <br /><pre>from functools import wraps<br />from google.appengine.api import users<br />from flask import redirect, request<br /><br />def login_required(func):<br /> @wraps(func)<br /> def decorated_view(*args, **kwargs):<br /> if not users.get_current_user():<br /> return redirect(users.create_login_url(request.url))<br /> return func(*args, **kwargs)<br /> return decorated_view<br /></pre>In the <em>new_post</em> view we will deal with forms. IMO, <a href="http://wtforms.simplecodes.com/" rel="nofollow" target="_blank">WTForms</a> is the best way to deal with forms in Flask. There is a Flask extension called Flask-WTF, and we can install it in our application for easy dealing with forms. Here is how can we install WTForms and Flask-WTF: <br /><pre>% wget http://pypi.python.org/packages/source/W/WTForms/WTForms-0.6.zip<br />% unzip WTForms-0.6.zip<br />% cp -r WTForms-0.6/wtforms ~/Projetos/blog/gaeseries/<br />% wget http://pypi.python.org/packages/source/F/Flask-WTF/Flask-WTF-0.2.3.tar.gz<br />% tar -xvzf Flask-WTF-0.2.3.tar.gz<br />% cp -r Flask-WTF-0.2.3/flaskext ~/Projetos/blog/gaeseries/<br /></pre>Now we have installed WTForms and Flask-WTF, and we can create a new WTForm with two fields: title and content. Remember that the date and author will be filled automatically with the current datetime and current user. Here is the <em>PostForm</em> code (I put it inside the <em>views.py</em> file, but it is possible to put it in a separated <em>forms.py</em> file): <br /><pre>from flaskext import wtf<br />from flaskext.wtf import validators<br /><br />class PostForm(wtf.Form):<br /> title = wtf.TextField('Title', validators=[validators.Required()])<br /> content = wtf.TextAreaField('Content', validators=[validators.Required()])<br /></pre>Now we can create the <em>new_post</em> view: <br /><pre>@app.route('/posts/new', methods = ['GET', 'POST'])<br />@login_required<br />def new_post():<br /> form = PostForm()<br /> if form.validate_on_submit():<br /> post = Post(title = form.title.data,<br /> content = form.content.data,<br /> author = users.get_current_user())<br /> post.put()<br /> flash('Post saved on database.')<br /> return redirect(url_for('list_posts'))<br /> return render_template('new_post.html', form=form)<br /></pre>Now, everything we need is to build the <em>new_post.html</em> template, here is the code for this template: <br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /> <h1 id="">Write a post</h1><br /> <form action="{{ url_for('new_post') }}" method="post" accept-charset="utf-8"><br /> {{ form.csrf_token }}<br /> <p><br /> <label for="title">{{ form.title.label }}</label><br /><br /> {{ form.title|safe }}<br /><br /> {% if form.title.errors %}<br /> <ul class="errors"><br /> {% for error in form.title.errors %}<br /> <li>{{ error }}</li><br /> {% endfor %}<br /> </ul><br /> {% endif %}<br /> </p><br /> <p><br /> <label for="content">{{ form.content.label }}</label><br /><br /> {{ form.content|safe }}<br /><br /> {% if form.content.errors %}<br /> <ul class="errors"><br /> {% for error in form.content.errors %}<br /> <li>{{ error }}</li><br /> {% endfor %}<br /> </ul><br /> {% endif %}<br /> </p><br /> <p><input type="submit" value="Save post"/></p><br /> </form><br />{% endblock %}<br /></pre>Now everything is working. We can run Google App Engine local development server and access the URL <em>http://localhost:8080/posts/new</em> on the browser, then write a post and save it! Everything is ready to deploy, and the deploy process is the same of web2py, just run on terminal: <br /><pre>% /usr/local/google_appengine/appcfg.py update .</pre>And now the application is online :) Check this out: <a href="http://3.latest.gaeseries.appspot.com/">http://3.latest.gaeseries.appspot.com</a> (use your Google Account to write posts).<br /><br />Y<span>ou can also check the code out in Github: </span><a href="https://github.com/fsouza/gaeseries/tree/flask">https://github.com/fsouza/gaeseries/tree/flask</a><span>.</span>fsouzanoreply@blogger.comhttp://f.souza.cc/Flying with web2py on Google App Enginetag:blogger.com,1999:blog-4820627057316560708.post-75032347491768718222019-12-10T03:42:20+00:00<img align="right" border="0" src="http://3.bp.blogspot.com/-8b3S1hMG8bY/T_21jnabfjI/AAAAAAAAA54/5WVH26ICzxM/s1600/web2py_logo.png" />Here is the second part of the series about Python frameworks under Google App Engine. Now we will talk about <a href="http://www.web2py.com/" rel="nofollow" target="_blank">web2py</a>, a simple and fast <a href="http://f.souza.cc/search/label/python">Python</a> web framework. Like <a href="http://f.souza.cc/search/label/django">Django</a>, web2py has a great data abstraction layer. Unlike Django, the web2py data abstraction layer (DAL) was designed to manage non-relational databases, including BigTable.<br /><a name="more"></a><br />The first step is setup the environment, which is something really easy ;) First, access the <a href="http://www.web2py.com/" rel="nofollow" target="_blank">web2py official website</a> and in download section, get the source code in a zip file called <i>web2py_src.zip</i>. After download this file, extract it. A directory called web2py will be created, I renamed it to <i>web2py_blog</i>, but it is not relevant. web2py extracted directory is ready to Google App Engine, it contains an app.yaml file with settings of the application, for the application developed here, the following file was used: <br /><pre>application: gaeseries<br />version: 2<br />api_version: 1<br />runtime: python<br /><br />handlers:<br /><br />- url: /(?P<a>.+?)/static/(?P<b>.+)<br /> static_files: applications/\1/static/\2<br /> upload: applications/(.+?)/static/(.+)<br /> secure: optional<br /> expiration: "90d"<br /><br />- url: /admin-gae/.*<br /> script: $PYTHON_LIB/google/appengine/ext/admin<br /> login: admin<br /> <br />- url: /_ah/queue/default<br /> script: gaehandler.py<br /> login: admin<br /><br />- url: .*<br /> script: gaehandler.py <br /> secure: optional<br /><br />skip_files: |<br />^(.*/)?(<br /> (app\.yaml)|<br /> (app\.yml)|<br /> (index\.yaml)|<br /> (index\.yml)|<br /> (#.*#)|<br /> (.*~)|<br /> (.*\.py[co])|<br /> (.*/RCS/.*)|<br /> (\..*)|<br /> ((admin|examples|welcome)\.tar)|<br /> (applications/(admin|examples)/.*)|<br /> (applications/.*?/databases/.*) |<br /> (applications/.*?/errors/.*)|<br /> (applications/.*?/cache/.*)|<br /> (applications/.*?/sessions/.*)|<br /> )$<br /></pre>I changed only the two first lines, everything else was provided by web2py. The web2py project contains a subdirectory called <i>applications</i> where the web2py applications are located. There is an application called <i>welcome</i> used as scaffold to build new applications. So, let’s copy this directory and rename it to <i>blog</i>. Now we can walk in the same way that we walked in the <a href="http://f.souza.cc/2010/08/flying-with-django-on-google-app-engine.html">django post</a>: we will use two actions on a controller: one protected by login, where we will save posts, and other public action, where we will list all posts.<br /><br />W<span>e need to define our table model using the web2py database abstraction layer. There is a directory called </span><i>models</i><span> with a file called </span><i>db.py</i><span> inside the application directory (</span><i>blog</i><span>). There are a lot of code in this file, and it is already configured to use Google App Engine (web2py is amazing here) and the web2py built-in authentication tool. We will just add our </span><i>Post</i><span> model at the end of the file. Here is the code that defines the model:</span><br /><pre>current_user_id = (auth.user and auth.user.id) or 0<br /><br />db.define_table('posts', db.Field('title'),<br /> db.Field('content', 'text'),<br /> db.Field('author', db.auth_user, default=current_user_id, writable=False),<br /> db.Field('date', 'datetime', default=request.now, writable=False)<br /> )<br /><br />db.posts.title.requires = IS_NOT_EMPTY()<br />db.posts.content.requires = IS_NOT_EMPTY()<br /></pre>This code looks a little strange, but it is very simple: we define a database table called posts with four fields: <i>title</i> (a <i>varchar</i> – default type), <i>content</i> (a <i>text</i>), <i>author</i> (a <strike>foreign key</strike> – forget this in BigTable – to the auth_user table) and <i>date</i> (an automatically filled <i>datetime</i> field). On the last two lines, we define two validations to this model: <i>title</i> and <i>content</i> should not be empty.<br /><br />Now is the time to define a controller with an action to list all posts registered in the database. Another subdirectory of the <i>blog</i> application is the controllers directory, where we put the controllers. web2py controllers are a Python module, and each function of this module is an action, which responds to HTTP requests. web2py has an automatic URL convention for the action: /<application>/<controller>/<action>. In our example, we will have a controller called posts, so it will be a file called <i>posts.py</i> inside the controllers directory.<br /><br /><span>In the controller <i>posts.py</i>, we will have the action <i>index</i>, in that way, when we access the URL <i>/blog/posts</i>, we will see the list of the posts. Here is the code of the <i>index</i> action:</span><br /><pre>def index():<br /> posts = db().select(db.posts.ALL)<br /> return response.render('posts/index.html', locals())<br /></pre>As you can see, is just a few of code :) Now we need to make the <i>posts/index.html</i> view. The web2py views system allow the developer to use native Python code on templates, what means that the developer/designer has more power and possibilities. Here is the code of the view<i> posts/index.html </i>(it should be inside the <i>views</i> directory): <br /><pre>{{extend 'layout.html'}}<br /><h1 id="">Listing all posts</h1><br /><dl><br /> {{for post in posts:}}<br /> <dt>{{=post.title}} (written by {{=post.author.first_name}})</dt><br /> <dd>{{=post.content}}</dd><br /> {{pass}}<br /></dl><br /></pre>And now we can run the Google App Engine server locally by typing the following command inside the project root (I have the Google App Engine SDK extracted on my <i>/usr/local/google_appengine</i>): <br /><pre>% /usr/local/google_appengine/dev_appserver.py .</pre>If you check the URL http://localhost:8080/blog/posts, then you will see that we have no posts in the database yet, so let’s create the login protected action that saves a post on the database. Here is the action code: <br /><pre>@auth.requires_login()<br />def new():<br /> form = SQLFORM(db.posts, fields=['title','content'])<br /> if form.accepts(request.vars, session):<br /> response.flash = 'Post saved.'<br /> redirect(URL('blog', 'posts', 'index'))<br /> return response.render('posts/new.html', dict(form=form))<br /></pre>Note that there is a decorator. web2py includes a complete authentication and authorization system, which includes an option for new users registries. So you can access the URL <i>/blog/default/user/register</i> and register yourself to write posts :) Here is the <i>posts/new.html</i> view code, that displays the form: <br /><pre>{{extend 'layout.html'}}<br /><br /><h1 id=""><br />Save a new post</h1><br />{{=form}}<br /></pre>After it the application is ready to the deploy. The way to do it is running the following command on the project root: <br /><pre>% /usr/local/google_appengine/appcfg.py update .</pre>And see the magic! :) You can check this application live here: <a href="http://2.latest.gaeseries.appspot.com/" target="_blank">http://2.latest.gaeseries.appspot.com/</a> (you can login with the e-mail demo@demo.com and the password demo, you can also register yourself).<br /><br />And the code here: <a href="https://github.com/fsouza/gaeseries/tree/web2py">https://github.com/fsouza/gaeseries/tree/web2py</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Using Juju to orchestrate CentOS-based cloud servicestag:blogger.com,1999:blog-4820627057316560708.post-76353590149121223122019-12-10T03:42:20+00:00<img align="right" border="0" height="157" src="https://4.bp.blogspot.com/-yl2WIzKZymo/UBQj4zeCl6I/AAAAAAAAA9U/otROJJC8aSQ/s400/juju.png" width="150" />Earlier this week I had the opportunity to meet <a href="https://twitter.com/kylemacdonald/" rel="nofollow" target="_blank">Kyle MacDonald</a>, head of <a href="http://cloud.ubuntu.com/" rel="nofollow" target="_blank">Ubuntu Cloud</a>, during <a href="http://fisl.org.br/13" rel="nofollow" target="_blank">FISL</a>, and he was surprised when we told him we are <a href="https://lists.ubuntu.com/archives/juju-dev/2012-June/000001.html" rel="nofollow" target="_blank">using Juju with CentOS</a> at Globo.com. Then I decided to write this post explaining how we came up with a <a href="https://github.com/globocom/juju-centos-6" rel="nofollow" target="_blank">patched version of Juju</a> that allows us to have CentOS clouds managed by Juju.<a name="more"></a><br /><br />For those who doesn't know Juju, it's a service orchestration tool, focused on <a href="http://en.wikipedia.org/wiki/DevOps" rel="nofollow" target="_blank">devops</a> "development method". It allows you to deploy services on clouds, local machine and even bare metal machines (using Canonical's MAAS). <br /><br />It's based on <a href="https://juju.ubuntu.com/docs/charm-store.html" rel="nofollow" target="_blank">charms</a> and very straightforward to use. Here is a very basic set of commands with which you can deploy a Wordpress related to a MySQL service: <pre><br />% juju bootstrap<br />% juju deploy mysql<br />% juju deploy wordpress<br />% juju add-relation wordpress mysql<br />% juju expose wordpress<br /></pre>These commands will boostrap the environment, setting up a bootstrap machine which will manage your services; deploy mysql and wordpress instances; add a relation between them; and expose the wordpress port. The voilà, we have a wordpress deployed, and ready to serve our posts. Amazing, huh? <br /><br />But there is an issue: although you can install the <code>juju</code> command line tool in almost any OS (including Mac OS), right now you are able do deploy only Ubuntu-based services (you must use an Ubuntu instance or container). <br /><br />To change this behavior, and enable Juju to spawn CentOS instances (and containers, if you have a CentOS lxc template), we need to develop and apply some changes to Juju and <a href="https://help.ubuntu.com/community/CloudInit" rel="nofollow" target="_blank">cloud-init</a>. Juju uses cloud-init to spawn machines with proper dependencies set up, and it's based on modules. All we need to do, is add a module able to install rpm packages using <code>yum</code>. <br /><br />cloud-init modules are Python modules that starts with <code>cc_</code> and implement a `handle` function (for example, a module called "yum_packages" would be written to a file called <code>cc_yum_packages.py</code>). So, here is the code for the module <code>yum_packages</code>: <pre><br />import subprocess<br />import traceback<br /><br />from cloudinit import CloudConfig, util<br /><br />frequency = CloudConfig.per_instance<br /><br /><br />def yum_install(packages):<br /> cmd = ["yum", "--quiet", "--assumeyes", "install"]<br /> cmd.extend(packages)<br /> subprocess.check_call(cmd)<br /><br /><br />def handle(_name, cfg, _cloud, log, args):<br /> pkglist = util.get_cfg_option_list_or_str(cfg, "packages", [])<br /><br /> if pkglist:<br /> try:<br /> yum_install(pkglist)<br /> except subprocess.CalledProcessError:<br /> log.warn("Failed to install yum packages: %s" % pkglist)<br /> log.debug(traceback.format_exc())<br /> raise<br /><br /> return True<br /></pre>The module installs all packages listed in cloud-init yaml file. If we want to install `emacs-nox` package, we would write this yaml file and use it as user data in the instance: <pre><br />#cloud-config<br />modules:<br /> - yum_packages<br />packages: [emacs-nox]<br /></pre>cloud-init already works on Fedora, with Python 2.7, but to work on CentOS 6, with Python 2.6, it needs a patch: <pre><br />--- cloudinit/util.py 2012-05-22 12:18:21.000000000 -0300<br />+++ cloudinit/util.py 2012-05-31 12:44:24.000000000 -0300<br />@@ -227,7 +227,7 @@<br /> stderr=subprocess.PIPE, stdin=subprocess.PIPE)<br /> out, err = sp.communicate(input_)<br /> if sp.returncode is not 0:<br />- raise subprocess.CalledProcessError(sp.returncode, args, (out, err))<br />+ raise subprocess.CalledProcessError(sp.returncode, args)<br /> return(out, err)<br /></pre>I've packet up this module and this patch in a <a href="https://github.com/globocom/cloudinit-centos-6" rel="nofollow" target="_blank">RPM package</a> that must be pre-installed in the lxc template and AMI images. Now, we need to change Juju in order to make it use the <code>yum_packages</code> module, and include all RPM packages that we need to install when the machine borns. <br /><br />Is Juju, there is a class that is responsible for building and rendering the YAML file used by cloud-init. We can extend it and change only two methods: <code>_collect_packages</code>, that returns the list of packages that will be installed in the machine after it is spawned; and <code>render</code> that returns the file itself. Here is our <code>CentOSCloudInit</code> class (within the patch): <pre><br />diff -u juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py juju-0.5-bzr531/juju/providers/common/cloudinit.py<br />--- juju-0.5-bzr531.orig/juju/providers/common/cloudinit.py 2012-05-31 15:42:17.480769486 -0300<br />+++ juju-0.5-bzr531/juju/providers/common/cloudinit.py 2012-05-31 15:55:13.342884919 -0300<br />@@ -324,3 +324,32 @@<br /> "machine-id": self._machine_id,<br /> "juju-provider-type": self._provider_type,<br /> "juju-zookeeper-hosts": self._join_zookeeper_hosts()}<br />+<br />+<br />+class CentOSCloudInit(CloudInit):<br />+<br />+ def _collect_packages(self):<br />+ packages = [<br />+ "bzr", "byobu", "tmux", "python-setuptools", "python-twisted",<br />+ "python-txaws", "python-zookeeper", "python-devel", "juju"]<br />+ if self._zookeeper:<br />+ packages.extend([<br />+ "zookeeper", "libzookeeper", "libzookeeper-devel"])<br />+ return packages<br />+<br />+ def render(self):<br />+ """Get content for a cloud-init file with appropriate specifications.<br />+<br />+ :rtype: str<br />+<br />+ :raises: :exc:`juju.errors.CloudInitError` if there isn't enough<br />+ information to create a useful cloud-init.<br />+ """<br />+ self._validate()<br />+ return format_cloud_init(<br />+ self._ssh_keys,<br />+ packages=self._collect_packages(),<br />+ repositories=self._collect_repositories(),<br />+ scripts=self._collect_scripts(),<br />+ data=self._collect_machine_data(),<br />+ modules=["ssh", "yum_packages", "runcmd"])<br /></pre>The other change we need is in the <code>format_cloud_init</code> function, in order to make it recognize the <code>modules</code> parameter that we used above, and tell cloud-init to not run <code>apt-get</code> (update nor upgrade). Here is the patch: <pre><br />diff -ur juju-0.5-bzr531.orig/juju/providers/common/utils.py juju-0.5-bzr531/juju/providers/common/utils.py<br />--- juju-0.5-bzr531.orig/juju/providers/common/utils.py 2012-05-31 15:42:17.480769486 -0300<br />+++ juju-0.5-bzr531/juju/providers/common/utils.py 2012-05-31 15:44:06.605014021 -0300<br />@@ -85,7 +85,7 @@<br /> <br /> <br /> def format_cloud_init(<br />- authorized_keys, packages=(), repositories=None, scripts=None, data=None):<br />+ authorized_keys, packages=(), repositories=None, scripts=None, data=None, modules=None):<br /> """Format a user-data cloud-init file.<br /> <br /> This will enable package installation, and ssh access, and script<br />@@ -117,8 +117,8 @@<br /> structure.<br /> """<br /> cloud_config = {<br />- "apt-update": True,<br />- "apt-upgrade": True,<br />+ "apt-update": False,<br />+ "apt-upgrade": False,<br /> "ssh_authorized_keys": authorized_keys,<br /> "packages": [],<br /> "output": {"all": "| tee -a /var/log/cloud-init-output.log"}}<br />@@ -136,6 +136,11 @@<br /> if scripts:<br /> cloud_config["runcmd"] = scripts<br /> <br />+ if modules:<br />+ cloud_config["modules"] = modules<br />+<br /> output = safe_dump(cloud_config)<br /> output = "#cloud-config\n%s" % (output)<br /> return output<br /></pre>This patch is also packed up within <a href="https://github.com/globocom/juju-centos-6" rel="nofollow" target="_blank">juju-centos-6</a> repository, which provides sources for building RPM packages for juju, and also some pre-built RPM packages. <br /><br />Now just build an AMI image with <code>cloudinit</code> pre-installed, configure your juju <code>environments.yaml</code> file to use this image in the environment and you are ready to deploy cloud services on CentOS machines using Juju! <br /><br />Some caveats: <ul><li>Juju needs a user called <code>ubuntu</code> to interact with its machines, so you will need to create this user in your CentOS AMI/template.</li><li>You need to host all RPM packages for <a href="https://github.com/globocom/juju-centos-6" rel="nofollow" target="_blank">juju</a>, <a href="https://github.com/globocom/cloudinit-centos-6" rel="nofollow" target="_blank">cloud-init</a> and following dependencies in some <code>yum</code> repository (I haven't submitted them to any public repository): <ul><li><a href="https://github.com/globocom/python-txaws-centos-6" rel="nofollow" target="_blank">python-txaws</a></li><li><a href="https://github.com/globocom/python-txzookeeper-centos-6" rel="nofollow" target="_blank">python-txzookeeper</a></li><li><a href="https://github.com/globocom/zookeeper-centos-6" rel="nofollow" target="_blank">zookeeper</a></li></ul></li><li>With this patched Juju, you will have a pure-centos cloud. It does not enable you to have multiple OSes in the same environment.</li></ul>It's important to notice that we are going to put some effort to make the Go version of juju born supporting multiple OSes, ideally through an interface that makes it extensible to any other OS, not Ubuntu and CentOS only.fsouzanoreply@blogger.comhttp://f.souza.cc/Go solution for the Dining philosophers problemtag:blogger.com,1999:blog-4820627057316560708.post-40607293978787369612019-12-10T03:42:20+00:00<img align="right" border="0" height="154" src="https://4.bp.blogspot.com/-Nh7xB1pO9UE/T_7ORn9Fa0I/AAAAAAAAA74/uLgLHM2QZUo/s400/dining-philosophers-problem.png" width="160" />I spent part of the sunday solving the <a href="http://en.wikipedia.org/wiki/Dining_philosophers_problem" target="_blank">Dining Philosophers</a> using <a href="http://golang.org/" target="_blank">Go</a>. The given solution is based in the description for the problem present in <a href="http://greenteapress.com/semaphores/" rel="nofollow" target="_blank">The Little Book of Semaphores</a>: <br /><br /><i>The Dining Philosophers Problem was proposed by Dijkstra in 1965, when dinosaurs ruled the earth. It appears in a number of variations, but the standard features are a table with five plates, five forks (or chopsticks) and a big bowl of spaghetti.</i><br /><br />There are some constraints: <br /><ul><li>Only one philosopher can hold a fork at a time</li><li>It must be impossible for a deadlock to occur</li><li>It must be impossible for a philosopher to starve waiting for a fork</li><li>It must be possible for more than one philosopher to eat at the same time</li></ul><a name="more"></a>No more talk, here is my solution for the problem: <br /><pre>package main<br /><br />import (<br /> "fmt"<br /> "sync"<br /> "time"<br />)<br /><br />type Fork struct {<br /> sync.Mutex<br />}<br /><br />type Table struct {<br /> philosophers chan Philosopher<br /> forks []*Fork<br />}<br /><br />func NewTable(forks int) *Table {<br /> t := new(Table)<br /> t.philosophers = make(chan Philosopher, forks - 1)<br /> t.forks = make([]*Fork, forks)<br /> for i := 0; i < forks; i++ {<br /> t.forks[i] = new(Fork)<br /> }<br /> return t<br />}<br /><br />func (t *Table) PushPhilosopher(p Philosopher) {<br /> p.table = t<br /> t.philosophers <- data-blogger-escaped-0="" data-blogger-escaped-1="" data-blogger-escaped-2="" data-blogger-escaped-3="" data-blogger-escaped-4="" data-blogger-escaped-:="range" data-blogger-escaped-_="" data-blogger-escaped-able="" data-blogger-escaped-anscombe="" data-blogger-escaped-artin="" data-blogger-escaped-chan="" data-blogger-escaped-e9="" data-blogger-escaped-eat="" data-blogger-escaped-eating...="" data-blogger-escaped-eter="" data-blogger-escaped-f="" data-blogger-escaped-fed.="" data-blogger-escaped-fed="" data-blogger-escaped-fmt.printf="" data-blogger-escaped-for="" data-blogger-escaped-func="" data-blogger-escaped-getforks="" data-blogger-escaped-go="" data-blogger-escaped-heidegger="" data-blogger-escaped-homas="" data-blogger-escaped-index="" data-blogger-escaped-int="" data-blogger-escaped-is="" data-blogger-escaped-leftfork.lock="" data-blogger-escaped-leftfork.unlock="" data-blogger-escaped-leftfork="" data-blogger-escaped-leibniz="" data-blogger-escaped-len="" data-blogger-escaped-lizabeth="" data-blogger-escaped-lombard="" data-blogger-escaped-main="" data-blogger-escaped-make="" data-blogger-escaped-n="" data-blogger-escaped-nagel="" data-blogger-escaped-name="" data-blogger-escaped-ork="" data-blogger-escaped-ottfried="" data-blogger-escaped-p.eat="" data-blogger-escaped-p.fed="" data-blogger-escaped-p.getforks="" data-blogger-escaped-p.name="" data-blogger-escaped-p.putforks="" data-blogger-escaped-p.table.popphilosopher="" data-blogger-escaped-p.table.pushphilosopher="" data-blogger-escaped-p.table="nil" data-blogger-escaped-p.think="" data-blogger-escaped-p="" data-blogger-escaped-philosopher="" data-blogger-escaped-philosopherindex="" data-blogger-escaped-philosophers="" data-blogger-escaped-popphilosopher="" data-blogger-escaped-pre="" data-blogger-escaped-putforks="" data-blogger-escaped-return="" data-blogger-escaped-rightfork.lock="" data-blogger-escaped-rightfork.unlock="" data-blogger-escaped-rightfork="" data-blogger-escaped-s="" data-blogger-escaped-string="" data-blogger-escaped-struct="" data-blogger-escaped-t.forks="" data-blogger-escaped-t="" data-blogger-escaped-table="" data-blogger-escaped-think="" data-blogger-escaped-thinking...="" data-blogger-escaped-time.sleep="" data-blogger-escaped-type="" data-blogger-escaped-was=""><br />Any feedback is very welcome.<!-----></pre>fsouzanoreply@blogger.comhttp://f.souza.cc/Speaking at PythonBrasil[7]tag:blogger.com,1999:blog-4820627057316560708.post-85363361206365246312019-12-10T03:42:20+00:00<img align="left" border="0" height="150" src="https://1.bp.blogspot.com/-YDAeiUQcCwk/T_5HsQXLsCI/AAAAAAAAA7o/W8loC-rAm84/s400/python-brasil.png" width="179" />Next weekend I’ll be talking about scaling Django applications at <a href="http://www.pythonbrasil.org.br/" rel="nofollow" target="_blank">Python Brasil</a>, the brazilian Python conference. It will be my first time at the conference, which is one of the greatest <a href="http://f.souza.cc/search/label/python" rel="nofollow" target="_blank">Python</a> conferences in Latin America. <br /><br />Some international dudes are also attending to the conference: <a href="http://cyberwebconsulting.com/" rel="nofollow" target="_blank">Wesley Chun</a> is going to talk about Python 3 and Google App Engine; <a href="http://www.enfoldsystems.com/" rel="nofollow" target="_blank">Alan Runyan</a> will talk about free and open source software, and <a href="http://holdenweb.com/" rel="nofollow" target="_blank">Steve Holden</a> will be talking about the issues involved in trying to build a global Python user group.<a name="more"></a><br /><br />There is also <a href="https://twitter.com/#!/fijall" rel="nofollow" target="_blank">Maciej Fijalkowski</a>, PyPy core developer, talking about little things PyPy makes possible. <br /><br />As I pointed before, I’m going to talk about scalability, based in some experiences aquired scaling Django applications at <a href="http://globo.com" rel="nofollow" target="_blank">Globo.com</a>, like <a href="http://g1.globo.com" rel="nofollow" target="_blank">G1</a>, the greatest news portal in the Latin America.fsouzanoreply@blogger.comhttp://f.souza.cc/Flying with Django on Google App Enginetag:blogger.com,1999:blog-4820627057316560708.post-18313231010347678152019-12-10T03:42:19+00:00<img align="right" border="0" src="http://2.bp.blogspot.com/-MWnn_9KeOMA/T_zRbAeW2WI/AAAAAAAAA5k/auY2uYcpwU4/s1600/gae_logo.png" /><a href="https://developers.google.com/appengine/" rel="nofollow">Google App Engine</a> is a powerful tool for web developers. I am sure that it is useful and every developer should taste it =) <a href="http://python.org/" rel="nofollow">Python</a> was the first programming language supported by App Engine, and is a programming language with a lot of web frameworks. So, you can use some of these frameworks on Google App Engine. In a series of three blog posts, I will show how to use three Python web frameworks on App Engine: <a href="http://djangoproject.com/" rel="nofollow">Django</a>, <a href="http://flask.pocoo.org/" rel="nofollow">Flask</a> and <a href="http://web2py.com/" rel="nofollow">web2py</a> (not necessarily in this order).<br /><a name="more"></a><div><br /></div><div>The first framework is Django, the most famous of all Python frameworks and maybe is used the most.</div><div><br /></div><div><div>Django models is the strongest Django feature. It is a high level database abstraction layer with a powerful object-relational mapper, it supports a lot of relational database management systems, but App Engine doesn’t use a relational database. The database behind App Engine is called BigTable, which is a distributed storage system for managing structured data, designed to scale to a very large size (Reference: <a href="http://labs.google.com/papers/bigtable.html" rel="nofollow">Bigtable: A Distributed Storage System for Structured Data</a>). It is not based on schemas, tables, keys or columns, it is like a big map indexed by a row key, column key and a timestamp. We can not use native version of Django models with Bigtable, because the Django models framework was not designed for non relational databases.</div></div><div><br /></div>So, what can we do? There is a Django fork, the <a href="http://www.allbuttonspressed.com/projects/django-nonrel">django-nonrel project</a>, which aims to bring the power of the Django model layer to non-relational databases. I will use the <a href="http://www.allbuttonspressed.com/projects/djangoappengine">djangoappengine</a> sub-project to build the sample application of this post, that will be deployed on Google App Engine :)<br /><div><br /></div><div><div>The sample application is the default: a blog. A very simple blog, with only a form protected by login (using Django built-in authentication system instead of Google Accounts API) and a public page listing all blog posts. It is very easy and simple to do, so let’s do it.</div><div><br /></div></div>First, we have to setup our environment. According the <a href="http://www.allbuttonspressed.com/projects/djangoappengine#installation">djangoappengine project documentation</a>, we need to download 4 zip files and put it together. First, I downloaded the django-testapp file, extract its contents and renamed the project directory from <i>django-testapp</i> to <i>blog_gae</i>. After this step, I downloaded the other files and put it inside the <i>blog_gae</i> directory. Here is the final project structure:<br /><div><br /></div><div class="separator"><img border="0" src="http://3.bp.blogspot.com/-ODEBZsiEJEY/T_zRi_PPD3I/AAAAAAAAA5s/tcxAI38pMhM/s1600/django_structure.png" /></div><div><br /></div>“django” directory is from the <i>django-nonrel</i> zip file, “djangoappengine” directory is from <i>djangoappengine</i> zip file and “djangotoolbox” directory is from <i>djangotoolbox</i> zip file. Look that is provided an <i>app.yaml</i> file, ready to be customized. I just changed the application id inside this file. The final code of the file is the following:<br /><pre>application: gaeseries<br />version: 1<br />runtime: python<br />api_version: 1<br /><br />default_expiration: '365d'<br /><br />handlers:<br />- url: /remote_api<br /> script: $PYTHON_LIB/google/appengine/ext/remote_api/handler.py<br /> login: admin<br /><br />- url: /_ah/queue/deferred<br /> script: djangoappengine/deferred/handler.py<br /> login: admin<br /><br />- url: /media/admin<br /> static_dir: django/contrib/admin/media/<br /><br />- url: /.*<br /> script: djangoappengine/main/main.py<br /></pre>I will use one version for each part of the series, so it is the first version because it is the first part =D In <i>settings.py</i>, we just uncomment the app <i>django.contrib.auth</i> line inside the <i>INSTALLED_APPS</i> tuple, because we want to use the built-in auth application instead of the Google Accounts API provided by App Engine.<br /><br />All settings are ok now, it is time to create the core application. In the Django project, we will use the core application to manage models and serve some views. We just start it using the following command:<br /><pre>% python manage.py startapp core</pre>It is a famous Django command, that creates the application structure, which is a Python package containing 3 Python modules: <em>models</em>, <em>tests</em> and <em>views</em>. Now we have to create the <em>Post</em> model. Here is the code of <em>models.py</em> file:<br /><pre>from django.db import models<br />from django.contrib.auth.models import User<br /><br />class Post(models.Model):<br /> title = models.CharField(max_length = 200)<br /> content = models.TextField()<br /> date = models.DateTimeField(auto_now_add = True)<br /> user = models.ForeignKey(User)<br /></pre>Now we just need to “install” the core application putting it on INSTALLED_APPS tuple in settings.py file and Django will be ready to play with BigTable. :) We will use the django.contrib.auth app, so let’s run a manage command to create a superuser:<br /><pre>% python manage.py createsuperuser</pre>After create the superuser, we need to setup login and logout URLs, and make two templates. So, in <i>urls.py</i> file, put two mappings to login and logout views. The file will look like this:<br /><pre>from django.conf.urls.defaults import *<br /><br />urlpatterns = patterns('',<br /> ('^$', 'django.views.generic.simple.direct_to_template',<br /> {'template': 'home.html'}),<br /><br /> ('^login/$', 'django.contrib.auth.views.login'),<br /> ('^logout/$', 'django.contrib.auth.views.logout'),<br />)<br /></pre>Here is the <i>registration/login.html</i> template:<br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /><br /><p>Fill the form below to login in the system ;)</p><br /><br />{% if form.errors %}<br /><p>Your username and password didn't match. Please try again.</p><br />{% endif %}<br /><br /><form method="post" action="{% url django.contrib.auth.views.login %}">{% csrf_token %}<br /><table><br /><tr><br /> <td>{{ form.username.label_tag }}</td><br /> <td>{{ form.username }}</td><br /></tr><br /><tr><br /> <td>{{ form.password.label_tag }}</td><br /> <td>{{ form.password }}</td><br /></tr><br /></table><br /><br /><input type="submit" value="login" /><br /><input type="hidden" name="next" value="{{ next }}" /><br /></form><br /><br />{% endblock %}</pre>And registration/logged_out.html template: <br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /> Bye :)<br />{% endblock %}<br /></pre>See the two added lines in highlight. In settings.py file, add three lines:<br /><pre>LOGIN_URL = '/login/'<br />LOGOUT_URL = '/logout/'<br />LOGIN_REDIRECT_URL = '/'<br /></pre>And we are ready to code =) Let’s create the login protected view, where we will write and save a new post. To do that, first we need to create a Django Form, to deal with the data. There are two fields in this form: title and content, when the form is submitted, the user property is filled with the current logged user and the date property is filled with the current time. So, here is the code of the ModelForm:<br /><pre>class PostForm(forms.ModelForm):<br /> class Meta:<br /> model = Post<br /> exclude = ('user',)<br /><br /> def save(self, user, commit = True):<br /> post = super(PostForm, self).save(commit = False)<br /> post.user = user<br /><br /> if commit:<br /> post.save()<br /><br /> return post<br /></pre>Here is the <i>views.py</i> file, with the two views (one “mocked up”, with a simple redirect):<br /><pre>from django.contrib.auth.decorators import login_required<br />from django.shortcuts import render_to_response<br />from django.template import RequestContext<br />from django.http import HttpResponseRedirect<br />from django.core.urlresolvers import reverse<br />from forms import PostForm<br /><br />@login_required<br />def new_post(request):<br /> form = PostForm()<br /> if request.method == 'POST':<br /> form = PostForm(request.POST)<br /> if form.is_valid():<br /> form.save(request.user)<br /> return HttpResponseRedirect(reverse('core.views.list_posts'))<br /> return render_to_response('new_post.html',<br /> locals(), context_instance=RequestContext(request)<br /> )<br /><br />def list_posts(request):<br /> return HttpResponseRedirect('/')</pre>There is only two steps to do to finally save posts on BigTable: map a URL for the views above and create the <i>new_post.html</i> template. Here is the mapping code:<br /><pre>('^posts/new/$', 'core.views.new_post'),<br />('^posts/$', 'core.views.list_posts'),</pre>And here is the template code:<br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /> <form action="{% url core.views.new_post %}" method="post" accept-charset="utf-8"><br /> {% csrf_token %}<br /> {{ form.as_p }}<br /> <p><input type="submit" value="Post!"/></p><br /> </form><br />{% endblock %}</pre>Now, we can run on terminal <i>./manage.py runserver</i> and access the URL <i>http://localhost:8000/posts/new</i> on the browser, see the form, fill it and save the post :D The last one step is list all posts in http://localhost:8000/posts/. The list_posts view is already mapped to the URL <i>/posts/</i>, so we just need to create the code of the view and a template to show the list of posts. Here is the view code:<br /><pre>def list_posts(request):<br /> posts = Post.objects.all()<br /> return render_to_response('list_posts.html',<br /> locals(), context_instance=RequestContext(request)<br /> )</pre>And the <i>list_posts.html</i> template code:<br /><pre>{% extends "base.html" %}<br /><br />{% block content %}<br /><dl><br /> {% for post in posts %}<br /> <dt>{{ post.title }} (written by {{ post.user.username }})</dt><br /> <dd>{{ post.content }}</dd><br /> {% endfor %}<br /></dl><br />{% endblock %}</pre>Finished? Not yet :) The application now is ready to deploy. How do we deploy it? Just one command:<br /><pre>% python manage.py deploy</pre>Done! Now, to use everything that we have just created on App Engine remote server, just create a super user in that server and enjoy:<br /><pre>% python manage.py remote createsuperuser</pre>You can check this application flying on Google App Engine: <a href="http://1.latest.gaeseries.appspot.com/" rel="nofollow">http://1.latest.gaeseries.appspot.com</a> (use demo for username and password in login page).<br /><br />You can check this application code out in Github: <a href="http://github.com/fsouza/gaeseries/tree/django">http://github.com/fsouza/gaeseries/tree/django</a>.fsouzanoreply@blogger.comhttp://f.souza.cc/Interoperability #rust2020tag:blog.luizirber.org,2019-12-01:/2019/12/01/rust-2020/2019-12-01T15:00:00+00:00<p>In January I wrote a <a href="https://blog.luizirber.org/2019/01/05/rust-2019/">post</a> for the Rust 2019 call for blogs.
The <a href="https://blog.rust-lang.org/2019/10/29/A-call-for-blogs-2020.html">2020 call</a> is aiming for an RFC and roadmap earlier this time,
so here is my 2020 post =]</p>
<h3>Last call review: what happened?</h3>
<h4>An attribute proc-macro like <code>#[wasm_bindgen]</code> but for FFI</h4>
<p>This sort of happened... because WebAssembly is growing =]</p>
<p>I was very excited when <a href="https://hacks.mozilla.org/2019/08/webassembly-interface-types/">Interface Types</a> showed up in August,
and while it is still very experimental it is moving fast and bringing saner
paths for interoperability than raw C FFIs.
David Beazley even point this at the end of his <a href="https://www.youtube.com/watch?v=r-A78RgMhZU">PyCon India keynote</a>,
talking about how easy is to get information out of a WebAssembly module
compared to what had to be done for SWIG.</p>
<p>This doesn't solve the problem where strict C compatibility is required,
or for platforms where a WebAssembly runtime is not available,
but I think it is a great solution for scientific software
(or, at least, for my use cases =]).</p>
<h4>"More -sys and Rust-like crates for interoperability with the larger ecosystems" and "More (bioinformatics) tools using Rust!"</h4>
<p>I did some of those this year (<a href="https://crates.io/crates/bbhash-sys">bbhash-sys</a> and <a href="https://crates.io/crates/mqf">mqf</a>),
and also found some great crates to use in my projects.
Rust is picking up steam in bioinformatics,
being used as the primary choice for high quality software
(like <a href="https://varlociraptor.github.io/">varlociraptor</a>,
or the many coming from <a href="https://github.com/10XGenomics/">10X Genomics</a>)
but it is still somewhat hard to find more details
(I mostly find it on Twitter,
and sometime Google Scholar alerts).
It would be great to start bringing this info together,
which leads to...</p>
<h4>"A place to find other scientists?"</h4>
<p>Hey, this one happened! <a href="https://twitter.com/algo_luca/status/1081966759048028162">Luca Palmieri</a> started a conversation on <a href="https://www.reddit.com/r/rust/comments/ae77gt/scientific_computingmachine_learning_do_we_want_a/">reddit</a> and
the <a href="https://discord.gg/EXTSq4v">#science-and-ai</a> Discord channel on the Rust community server was born!
I think it works pretty well,
and Luca also has being doing a great job running <a href="https://github.com/LukeMathWalker/ndarray-koans">workshops</a>
and guiding the conversation around <a href="https://github.com/rust-ml/discussion">rust-ml</a>.</p>
<h2>Rust 2021: Interoperability</h2>
<p>Rust is amazing because it is very good at bringing many concepts and ideas that
seem contradictory at first,
but can really shine when <a href="https://rust-lang.github.io/rustconf-2018-keynote/#127">synthesized</a>.
But can we share this combined wisdom and also improve the situation in other
places too?
Despite the "Rewrite it in Rust" meme,
increased interoperability is something that is already driving a lot of the
best aspects of Rust:</p>
<ul>
<li>
<p>Interoperability with other languages: as I said before,
with WebAssembly (and Rust being having the best toolchain for it)
there is a clear route to achieve this,
but it will not replace all the software that already exist and can benefit
from FFI and C compatibility.
Bringing together developers from the many language specific binding
generators (<a href="https://github.com/tildeio/helix">helix</a>, <a href="https://github.com/neon-bindings/neon">neon</a>, <a href="https://github.com/rusterlium/rustler">rustler</a>, <a href="https://github.com/PyO3/pyo3">PyO3</a>...) and figuring out what's missing from
them (or what is the common parts that can be shared) also seems productive.</p>
</li>
<li>
<p>Interoperability with new and unexplored domains.
I think Rust benefits enormously from not focusing only in one domain,
and choosing to prioritize CLI, WebAssembly, Networking and Embedded is a good
subset to start tackling problems,
but how to guide other domains to also use Rust and come up with new
contributors and expose missing pieces of the larger picture?</p>
</li>
</ul>
<p>Another point extremely close to interoperability is training.
A great way to interoperate with other languages and domains is having good
documentation and material from transitioning into Rust without having to figure
everything at once.
Rust documentation is already amazing,
especially considering the many books published by each working group.
But... there is a gap on the transitions,
both from understanding the basics of the language and using it,
to the progression from beginner to intermediate and expert.</p>
<p>I see good resources for <a href="https://github.com/yoshuawuyts/rust-for-js-people">JavaScript</a> and <a href="https://github.com/rochacbruno/py2rs">Python</a> developers,
but we are still covering a pretty small niche:
programmers curious enough to go learn another language,
or looking for solutions for problems in their current language.</p>
<p>Can we bring more people into Rust?
<a href="https://rustbridge.com/">RustBridge</a> is obviously the reference here,
but there is space for much,
much more.
Using Rust in <a href="https://carpentries.org/">The Carpentries</a> lessons?
Creating <code>RustOpenSci</code>,
mirroring the communities of practice of <a href="https://ropensci.org/about/">rOpenSci</a> and <a href="https://www.pyopensci.org/">pyOpenSci</a>?</p>
<h2>Comments?</h2>
<ul>
<li><a href="https://social.lasanha.org/@luizirber/103236549475802733">Thread on Mastodon</a></li>
<li><a href="https://twitter.com/luizirber/status/1201373423592562690">Thread on Twitter</a></li>
</ul>luizirberhttps://blog.luizirber.org/