PySnippets – improving code reuse

For a long time now, I've been hindered by the issue of utilities, or snippets. These are convenience functions and classes that are too small or too incomplete to justify a library, yet are useful enough to be used.
I've posted a few on my blog: Namespaces, X and now FileDict. Others I didn't post, and include a priority queue, an A* implementation, a lazy-list, an LRU memoizer, etc. You probably have a few of those. I know because I see them on snippet sites.

However, I rarely actually use these in my code. I really want to. But once your code spans more than one file, you usually need to make a proper installation, or at least trouble your "users" a bit more. Usually saving a few lines just isn't worth the trouble. Copy-pasting the snippet into the file is sometimes the solution, but it really pains me that I'll have to re-paste it every time I improve my snippet.

I'm sure some of you looked at my snippets, or other people's, thought "cool", but never used them, simply because it was too much trouble.

Paradoxically, this is especially true when writing snippets. They are just one small file, and using another snippet would probably make them too hard to distribute. This is a magic-circle, for-ever limiting our snippets to a low level of sophistication, and discouraging re-use.

I want to break that circle. I want to create an economy of snippets, increasingly building on each other, eventually creating a "standard library" of their own. But how would one do that? I have one suggestion, along with a proof-of-concept, which I will present here.

PySnippets

PySnippets is my attempt of making snippets usable. It's comprised of two solutions - a server and a client.

  1. Server - A website for uploading snippets. Simple enough. You can rate them, tag them, discuss them, offer some documentation and of-course post newer versions.
  2. Client - A python module that automagically imports snippets from the web. Essentially, it downloads the snippets you request to a cache, imports them if they're already there, and periodically searches for updates.

The server is structured in a predictable way, so that the client knows how to fetch a snippet just by its name.

The Client

Here's a usage example with my current client implementation, I creatively call "snippets":

import snippets
antigravity = snippets.get('antigravity')  # "snippet-import"
antigravity.start(mode='xkcd')

Easy as that!

The snippets.get function looks for the module in the local snippets-cache. If it's there, get just imports it and returns the module. If it's not, it queries the server for a snippet called "antigravity" (names are unique), stores it in the cache, and the imports it. What the user notices is a 2-second pause the first time he ever imports that snippet, and nothing else from then on.

You can specify to download a specific version, like this:

filedict = snippets.get('filedict', version=0.1)

Auto-Updating Snippets

The current implementation also includes an "auto-update" feature: Periodically, before importing a module, the client surveys the server for a newer version of it. If a newer version exists, it downloads it to the cache and continues with the import.

Auto-updates can be disabled in a parameter to get.

The Server

The server is yet another service to upload snippets, however it has a slightly unusual design (which no other snippet site I know of has):

  • A URL to a snippet is easy to deduce given its name.
  • There is a conscious (though simple) support for versions.
  • To increase reliability and trust (more on that later), uploaded snippets cannot be altered (but a new version can be issued)

Since I know very little about administration and server-maintenance, I chose wikidot.com to host my POC web-site. They have an elaborate support for permissions and most of the features I need, such as the ability to rate, tag and discuss snippets.

Trust

Perhaps the biggest issue with such a system is trust. Since you're running code which resides online, you have to trust me not to maliciously alter the snippets, and also you have to trust the author of the snippet not to do so.

As a partial solution, uploaded files cannot be altered: Not edited, nor renamed, nor deleted, etc. So if specify a particular snippet version, it is guaranteed that it will never change (I may commit changes by request, but I will audit them myself).
If you decide to use the latest version of a snippet (that is, not specify a version), please make sure you trust its author.

Perhaps higher-ups in the Python community would like to take some sponsorship of the project, removing the remaining trust-issues with the administrator (that's me).

Implications

  • To distribute your snippets, all you need is for the reciever to have an internet connection, and the snippets client.
  • If you're sending someone code, you can attach the client (it's rather small, too), and just import away. The reciever will benefit from improvements and bugfixes to your snippets.
  • You can use other people's snippets just as easily, as long as you trust them.
  • Snippets can now build on each other without worrying too much.

What if my user is offline?

Then probably PySnippets isn't for him.

However, I do have some ideas, and might implement them if there is sufficient demand.

Afterword

PySnippets is my humble attempt at solving the utility/snippet reuse problems. I hope you like it and find it useful.

Please try it!

Tags: , , , , ,

Categorised in: ,

5 Comments

  • Omer Zak says:

    A better design (and a system which I might want to use) would be as follows.
    1. Follow the metaphor of compilations and 'make'.
    2. Whenever a project wants to use a snippet, define a compilation phase for it. In addition to compiling any C/C++ files or whatever, have the project's Makefile check for newer versions of snippets and if any - download them.
    3. Then it is possible to locally version-control, lock, review locally the snippets, and otherwise control whatever goes into the project - as well as enjoy improvements.
    4. Useful tools:
    - A command, invokable from Makefiles, to check if a more recent version of a file exists in a Website, and if yes - download it.
    - A way to automatically create list of files to be downloaded from imports in Python scripts.

    • erezsh says:

      Compilations fail.
      Unfortunately, C projects are not always portable, and different compilers accept different syntaxes and structures.
      Just in the past week easy_install failed three times on me because mingw32 couldn't compile the package.
      This has to Just Work, or it will be useless. Unless you suggest that I include my own compiler, I'm afraid C compilation cannot be included.

      Other than that, I'm not sure I'm following you. Please be more specific as to how what you suggest would improve on the current system.
      Thanks.

  • Omer Zak says:

    In the far future, the following could be handled by the setup.py package. For now, pure Python projects (software which consists only of *.py files) could have a special module, which downloads updated versions to all code snippets used by the project, when activated.

    The important point is that the snippets are not downloaded whenever the software is run but only when the software developer/user explicitly requests to update the snippets used by his software.

    This is like compilation of C/C++ which is performed only at request or updating of packages in common Linux distributions (such as Debian or Fedora). It is unlike the compilation of *.py into *.pyc, which is performed automatically.

  • Omer Zak says:

    Another thought:
    It would be nice to have a personal snippets library (could even be language-agnostic) with a library manager.

    Any project which uses a snippet, which could be shared with other projects (Python, shell, C++, whatever) would import the snippet from the library i.e. make a copy and register a link to the original in the library, to allow the copy to be automatically compared with the most recent version in the library.

    Whenever the project developer wants to, the library manager will review the snippets used in a project and propose updates for any snippet which was updated in the library.

    This will solve the vexing problem of project A having developed a snippet S, which is then used by project B - and improved due to the work on project B. We then want an easy way to get project A to use the improved S but only when we have the time to verify that the improved S didn't break project A.

    The personal snippets library could also act as a cache to the Internet-based snippets library.

Leave a Reply

Your email address will not be published. Required fields are marked *