Symphony.

@remie, as I coined before: moving to XML isn't going to magically solve all problems. Sure, you'll be able to merge the installations easier using diff, but more fundamental problems will still exist. In my opinion it's an illusion we can always automatically merge the two installations.

Instead, there will be cases where interaction from the user is required. In pretty much every project where multi-user CDI is important, will there be a "lead developer" who's in charge of getting the end-product working. If there is an easy way for him/her to merge changes and solutions by different developers, it's not really a problem that not everything is merged automatically.

Think of git merge here. If git was designed to automatically merge every request, it would have to know the spec of every project it is ever deployed on. It needs to be able to speak every language to know if merging A and B isn't going to produce a bug not present in A nor B. This is an impossible task, so Linus said: if there is a guarantee it will work, we will auto-merge. If there is a chance of a fuckup, let the user do it.

I think the same approach would work for Symphony, too. In fact, the UI can be a very good starting point: the auto-merge doesn't even have to do anything yet, it's just not that important. It's just a convenience.

So, long story short: start with a merge UI, then work on the auto-merge functionality. The extension will be usable from the start, and use-cases we've never thought about will arise soon enough (which is a big problem if you work the other way around).

Agreed, but I think it is easier to create such merge extension based on having the structure in XML rather than working with dozens of SQL statements which are not grouped and do not give a good overview of the intended action and dealing with auto-increment problems.

So either we need a meta language in which actions are defined or we need to store the changes structure in XML.

After the 2.3.1 release date, all efforts on the integration branch are for feature implementations in 2.4. Outstanding bugs and issues will more than likely be moved over to the new milestone too.

I will be helping Brendan split out the work to whoever is willing to do it, but the xml structure and config migration is the first on the list, as they both follow the same vein.

I think it is easier to create such merge extension based on having the structure in XML

Probably not. The thing is that the XML will only give the end-result, not the actions themselves. So if a section is renamed, the XML will not reflect this, and the extension will automatically duplicate the section. It's the same reason you are currently logging queries, and not comparing complete dumps.

edit: the renaming example didn't come out the way I wanted. If dev Alice renames section A to section B, her XML will contain section B, with the fields in it. If dev Bob started with the same sections, and left them in place, he will still have section A. Any automatic merge system will say: ah, these two sections have nothing in common: create both.

So, to get this properly working you'd need to build a transaction log (like you are doing with the CDI right now) anyway. If the end-result is stored in XML or SQL doesn't really matter, as you won't use it anyway.

edit: to conclude: if storing the transactions in XML is beneficial, by all means, do so. Yet this has pretty much nothing to do with how the core stores the data. So, waiting for the core to change this doesn't really make sense.

Any automatic merge system will say: ah, these two sections have nothing in common: create both.

Would a section's ID being stored in the XML help with such merging to a certain extent? I.e. the merge system could infer that there should only be one section per ID, and so knows that these two differing sections need to be merged into one.

Would a section's ID being stored in the XML help with such merging to a certain extent?

Unless it knew which one came first: nope. It would help with flagging the section for human intervention, but it wouldn't help with the merge itself.

The thing is that the XML will only give the end-result, not the actions themselves. So if a section is renamed, the XML will not reflect this, and the extension will automatically duplicate the section. It's the same reason you are currently logging queries, and not comparing complete dumps.

The point is that it is easier to add a hook to the structure changes in XML. Because of the relational structure we have in SQL, the number of queries involved with a single structure change is way bigger than the change itself. Listening to SQL changes, parsing them and figuring out how to deal with stuff like primary keys with auto-numbers and the fact that there can be multiple SQL statements handled in the same PHP script execution, makes working with SQL really hard.

if storing the transactions in XML is beneficial, by all means, do so. Yet this has pretty much nothing to do with how the core stores the data.

Catching changes to an XML structure file is easier. That way you can make an extension that listens to the changes to this file, create a change log based on meta-information and support the merging operation.

I've been trying to get this to work with the SQL statements, but I find al sorts of difficulties that I believe will not exist after moving to XML.

Hmm, I think that if you are logging the actions purely on the queries executed, you're doing it wrong. For every structural action, there is a delegate in the core, that you can just plug into.

This means you don't have to monitor queries, or files, you can just let Symphony tell you when things change.

So, it's relatively easy to hook into the SectionPostEdit delegate, and write the corresponding entry to the log, without having to be concerned about how this is then done in the core. Then, because all the data is present, you can just use SectionManager::edit with the data you logged to process the change on the master server.

Let Symphony do all the hard work (storing, checking, etc). Just log what happens, and call these functions with the same parameters when rebuilding.

Let me add to this that logging the actions does not solve the problem with primary keys by itself, but it does it every object (section, field, page, ds, event) is given a guid in the logs, which is not the same as its id in Symphony.

Then all relations are logged as relations between guid's, that are remapped to id's when the structure is rebuilt on the target machine. I agree this isn't easy at all, but I do think it's a lot more robust than polling for changes in a file, and 'guessing' what caused the differences.

This means you don't have to monitor queries, or files, you can just let Symphony tell you when things change.

Are you sure this covers all kind of changes you want to share? Right now, CDI also logs changes to extensions (enable, uninstall, etc).

Are you sure this covers all kind of changes you want to share?

Well, it should. This is pretty much exactly what the delegates are designed for, so if there is one missing, it's a bug.

Hey guys, just a quick question on what the ideal workflow with CDI looks like...

Right now I am doing it the following way
1. added the manifest/cdi to git
2. export the database on my dev site
3. push with git and pull on my prod site
4. Use the restore button on the database backup

Now I feel that isn't quite the way you inteded it to be, but I can't find any "The last 5 queries executed" on my slave after a git pull. Could you please list the exact workflow that you use?

Thank you!

Hmm not quite I think its more like.

1. Add manifest/cdi to git
2. Set your DEV to be Master in the CDI preferences
3. Set your production to be a Slave in the CDI preferences
4. Edit sections (for example)
5. Check that manifest/cdi/cdi.sql exists & contains queries
6. Push the cdi.sql through git
7. Go to preferences on production (and you will see x queries to execute) there is a link and you just click that and all queries should run automatically.

The problem is I can't see the "x queries to execute", and I do have data in cdi.sql, check my screenshot, that's all I see :(

Attachments:

Screen Shot 2012-09-24 at 08.56.46.png

That looks nearly good except that you should have a line on top of instance mode saying

Synchronize CDI Slave

with a link to something like yoursite.com/symphony/extension/cdi/update/24c17443b2

not sure what happens if you try without the hash code at the back. It says no queries executed because you never ran the update script before so that's presumably normal.

@ChriZ: could you try disabling and enabling the extension on your SLAVE instance (production)? You should indeed see the link that gunglien copied, that allows you to execute the CDI update on the slave (this is not done automatically).

Also, can you verify that the manifest/cdi/cdi.sql file is available on production?

not sure what happens if you try without the hash code at the back.

Nothing. The hash is there because the site will be unavailable during the insertion of new queries (maintenance mode will be enabled), so if an attacker repeatedly reloads that page, a very cheap DOS attack is born. Hence the hash.

@Huib: that is not entirely true... if you remove the hash from the config.php you will still be able to execute the update script. We forgot to add a check for an empty config setting :)

okay cdi.sql clearly is there... it also has some stuff in it, I checked the ownership etc. it's the same as the files in the export directory were, and they were listed...

running the script manually did something, so that seems to work!

also i deinstalled cdi and reinstalled it, no luck... :/ I'll take a look at the mysql tables and check if there are any duplicates (shouldn't be possible though right?!)

edit: just so we're on the same page, the cdi.sql is in manifest/cdi/cdi.sql right?

edit again: I'm using 1.0.1 for symphony 2.2.5

Attachments:

Screen Shot 2012-09-24 at 14.38.31.png

okay great, I got it to work by manually calling the url (without the hash).

Would be great to have handy link though... any ideas on that? Is it not available on 1.0.1? Also, how can I define a hash?

Thanks a lot guys! This sure is a great extension!