Ok, so this is the detailed version of the previous post.
To recap; the structure generation code is still missing a vital piece - the canonical checking. I have been implementing Jean-Loup Faulon's algorithm for generation, but there is no precise algorithm given for canonical checking. Here the relevant paragraph from the enumeration paper:
"Checking for canonicity is a common procedure of orderly enumeration algorithms, the procedure guarantees that the graphs generated are nonisomorphic. ...To verify that a graph is canonical, one labels the vertices of the graph in all possible ways. The graph is canonical if the initial labeling leads to a list of edges that is lexicographically smaller than the lists obtained with all other labelings. In the present paper, we have implemented two algorithms to verify canonicity, Tarjan tree canonization algorithm if the tested graph is acyclic and McKay’s Nauty technique otherwise"
Okay, so I don't understand this for a couple of reasons : firstly, 'labelling all possible ways' sounds like an n-factorial (n!) operation; secondly, nauty does not lexicographically compare edge strings (as far as I know). Sadly, I know that I have misunderstood something, since my code doesn't work, and theirs does. :(
The technique used by nauty is also used by bliss and saucy and is known as iterative refinement of partitions. A 'partition' is a division of a set into subsets called 'cells', and refinement of a partition roughly means making a finer partition that has cells at least as small, if not smaller that the original. The end of the refinement process are 'discrete' partitions that are the same as permutations, since each cell has only one member. This is sketched in the image below:
A simple implementation of a partition refiner is in my repository. It is a slightly modified version of the algorithm from a book I've mentioned before (CAGES). It tries to deal with vertex and edge colours, although I don't think that it does so all that well. However, for simple graphs, it does indeed produce the automorphism group, as well as check for a canonical graph.
For example, the simple 5-cycle graph in the image might be canonical (or 'canonically labelled') if the refinement process produces the identity permutation as the first discrete partition. This will depend on the particular cell selection algorithm, and the choice made of how to arrange newly split cells.
The modifications I made to the CAGES refiner were done to align the canonical checking with the graph enumeration process. This is important, as edges are added to a graph in a characteristic order, and this has to produce graphs that will be accepted as canonical. This is illustrated in the following image:
The cycle graphs are laid out in a standard way on the left, and in a linear view on the right. The linear view emphasises the order in which the edges might be added. In the canonical scheme I have chosen, vertices are connected to the next possible partner. To put it another way, the resulting graphs will have a minimal edge 'length' when laid out as a linear graph.
What the partition refinement process is doing is the same as labelling all possible ways. The leaves of the refinement tree are permutations. Assuming the refinement process gives automorphic permutations, this is almost the same as labelling within the orbits of the atoms. It seems like partitioning the atoms by signature, then refining this partition should work, but it doesn't.
Any better suggestions or clarifications are very welcome :)