Thursday, July 28, 2011

Sub-second precision is not enough

Turns out relying on st_mtime having sub-second precision is not reliable enough, as this small test demonstrates:
$ touch a b ; stat a b | grep Modify
Modify: 2011-07-28 15:36:19.505160175 +0300
Modify: 2011-07-28 15:36:19.505160175 +0300
Opening one more process seems to delay enough to show a difference:
$ touch a ; touch b ; stat a b | grep Modify
Modify: 2011-07-28 15:37:50.082659665 +0300
Modify: 2011-07-28 15:37:50.085158931 +0300
(test was done on a Debian Squeeze, ext4 fs).

Not to mention that Python's underlying type for floats is only good for around sixteen digits. Taking out the ten digits before the decimal point, that leaves about six for sub-seconds.

This test:
import os

diffs = 0

for i in range(10000):
    f = open('x', 'w+')
    f.close()

    one = os.stat('x').st_mtime

    f = open('x', 'w+')
    f.close()

    two = os.stat('x').st_mtime

    if one != two:
        diffs += 1

print diffs, 'diffs out of 10000 iterations'
is giving me around ~30 per run, which is obviously not good enough.

So the filecache decorator resorts to relying solely on st_ino and Mercurial's append only strategy to ensure the cache is reliable.

Wednesday, July 20, 2011

the filecache decorator

The past week or so I've been working on a new decorator that tracks files under the .hg/ directory for changes.

In short, you use it on a method and it turns it to a property with caching the result, like propertycache. But it also gives you the ability to invalidate the cached property, which triggers a stat(2) call that checks if the file behind the property changed since the last time it was read.

I used it on the dirstate, changelog, manifest, bookmark files, and the tags cache in localrepo so far. What it means in practice is that a call to repo.invalidate() is significantly cheaper where some of the above haven't changed since the last time they were read.

This is crucial so the command server's cached repository stays up-to-date where it changes by a different process than the server itself, i.e. via committing to it directly on the command line.

The main issue with this approach is that we fail if we end up missing changes. For example, a filesystem that doesn't have subsecond precision, will cause our cache to lie in the following situation:

time  action
0       file x is modified
0.1    file x is read, inserted to the cache
0.2    file x is modified again, size remains the same

We end up with file x from 0 in our cache. Now suppose we invalidate the cache, this triggers a stat('x'), in which st_mtime == 0, which according to our cache is the most recent version of x, hence no need to reread. But it was in fact modified afterwards, but our filesystem doesn't have the necessary precision to help us spot it.

So we have to make sure our cache is reliable, and if we can't, we must fallback to reading the file every time the cache is invalidated.

Luckily Mercurial's approach to writing files helps us here. Essentially most of the important files under .hg/ are either: 1) atomically replaced, 2) appended.

If our filesystem is able to tell us a) if a file is replaced, or b) if it has subsecond precision, we're basically good to go. Because if we have (a) then (1) is covered, and (2) is covered because st_size changes on append. And if we have (b) it's obvious.

The current plan is to use the above test to make sure our cache is reliable, otherwise read the file every time. In the future we can improve this by also noting when the file was read.

Friday, July 15, 2011

Mercurial libraries

It's been a few weeks since 1.9 was released and a few libraries that use the command server are starting to appear:

  • JavaHg by the guys at aragost
  • a .NET library written by Tak
  • python-hglib written by myself (not public yet)

Watch the wiki page for updates.

Sunday, June 26, 2011

Command Server client example

Now that the Command Server is part of core Mercurial (and will be released with 1.9), an example client was added to the wiki. It's pretty minimal and doesn't do anything fancier than running the command passed to it on the command line.

import sys, struct, subprocess

# connect to the server
server = subprocess.Popen(['hg', 'serve', '--cmdserver', 'pipe'],
                          stdin=subprocess.PIPE, stdout=subprocess.PIPE)

def readchannel(server):
    channel, length = struct.unpack('>cI', server.stdout.read(5))
    if channel in 'IL': # input
        return channel, length
    return channel, server.stdout.read(length)

def writeblock(data):
    server.stdin.write(struct.pack('>l', len(data)))
    server.stdin.write(data)
    server.stdin.flush()

# read the hello block
hello = readchannel(server)
print "hello block:", repr(hello)

# write the command
server.stdin.write('runcommand\n')
writeblock('\0'.join(sys.argv[1:]))

# receive the response
while True:
    channel, val = readchannel(server)
    if channel == 'o':
        print "output:", repr(val)
    elif channel == 'e':
        print "error:", repr(val)
    elif channel == 'r':
        print "exit code:", struct.unpack(">l", val)[0]
        break
    elif channel == 'L':
        print "(line read request)"
        writeblock(sys.stdin.readline(val))
    elif channel == 'I':
        print "(block read request)"
        writeblock(sys.stdin.read(val))
    else:
        print "unexpected channel:", channel, val
        if channel == channel.upper(): # required?
            break

# shut down the server
server.stdin.close()

Tuesday, June 14, 2011

Command Server Protocol

A couple of days ago I sent out the command server protocol to the list. Today I've implemented all of the comments made on that thread, with plenty of other refinements to the patch queue.

I've also updated the wiki page to reflect the current state of things. The protocol is looking good so far, with most of it pretty much nailed down.

1.9 code freeze is imminent (June 17th), so in the next few days I'll be doing my best to get an early version of the command server in.

Saturday, June 4, 2011

First PoC

Yesterday I sent out an early version of the command server to the mailing list (patch queue), which included:

  • new option --cmdserver to `hg serve`.
  • a small Python wrapper, hglib, around the server that can connect to a repository and run commands. Also included is a sample of how real hg commands might look like in the lib, see status().
  • a shell that uses hglib and runs commands against a given repository.
  • last 2 patches are an attempt to integrate the command server to the test suite.

In the next few days I'll be working on:
  • Support for interactive commands.
  • Making more tests pass the testsuite using the command server.
  • Finalizing the command wire protocol and writing it down in a wiki page.
  • Feedback from other devs on the PoC.

Wednesday, May 25, 2011

Let the coding begin

Yesterday I sent out my latest patch queue with several MQ improvements, among others a qrefresh --interactive. Now that those are out of the way, my main focus is shifted to my project (although I'm sure I'll continue sending other related patches during this period, time permitting).

The first thing to take care of is changing the dispatch module to allow an initialized repo object to be passed in -- this will be used by the command server since it will cache them for performance.