Import/export CSV
This is an open discussion with 142 replies, filed under Extensions.
Search
Seems like memory is kind of an issue with big sets of data.
I don't understand much of where this limitation is caused.
Is this dependent on what server you're on? Or is it php limitation?
Does symphony need more memory then other systems?
It's all about the memory limits set in the PHP ini file, also script timeouts could affect it too, but this specific problem is just the memory.
Look for memory_limit
at around line 233
in a standard PHP ini file with comments in. I've had to set this very high at times to run loads of data, sometimes at 1Gb
!
To my knowledge you can also increase the memory limit in your .htaccess
file:
php_value memory_limit 1G
@designermonkey: i couldn't find that file...
@phoque: That simply did the trick! Thank you!!
Why is this setting not standard in symphony???
Why is this setting not standard in symphony???
Because you might risk killing your server entirely if you allowed each and everyone to allocate as much memory as they liked. So please undo that setting after finishing your import. :-)
Ok, but i am pretty happy i found out about this, i even managed to run the export ensemble, which had memory problems as well... seems like memory issues are history now, untill i kill my server :)
I believe the memory issue in this case (and perhaps other extensions experience this as well) is because EntryManager->fetch()
fetches al entries and puts them into one big array. Not sure on how to overcome this, perhaps seperate calls to EntryManager->fetch()
could do the trick, but I don't have time for this any soon to look into. As usual, most of my extensions only get updated and put to the test as soon as clients start complaining about them ;-)
@Giel,
I know you only update your extensions when a client askes for it,
But i have a feature request that you could maybe add in the future..
Or if it's not too hard and you can put me in the right direction i will give it a try
I think a really helpfull feature would be to filter the export cv by a given field
For example; i have a catalog containing a handfull of different brandname's. They are defined in a field 'brand'.
So it would be cool if i could only export the records containing a certain brand.
I guess a parameter could take care of this something like this:
/symphony/extension/importcsv/?export§ion-export=9&filerfield=brand&value=branda
You could then even provide a direct downloadlink on the front-end for all the different brands.
I'd like to know your thoughts on the possibilities on this feature.
By the way, thanks so much for making this extension. I couldn't have dealt with all the data without it.
I am getting the following error on the System > Import/Export CSV page:
Fatal error: Call to a member function appendChild() on a non-object in /home/site/public_html/designingforthedivide.org/extensions/importcsv/content/content.index.php on line 29
Any help?
Have you downloaded the correct version? The master is for the upcoming Symphony 2.3, if your not beta-testing that one, you should download the 2.2.5-tag instead.
For anyone who is as green to Git as I am, to download a particular tag you use the commands:
git clone git://github.com/kanduvisla/importcsv.git cd importcsv/ git checkout 2.2.5
Since line breaks in textareas were messing up my CSVs and since I don't need the text area data in my CSV, I created a driver for this Extension which enters null values for textareas.
https://gist.github.com/1844627
Disclaimer: Its a hack of an existing driver and I don't really understand the PHP
What I'd really like to do is to leave all text areas out of the CSV altogether so that I don't get the empty columns. Can anyone suggest a solution?
Better still, it would be great to be able to set exceptions on a per-section basis so that sub-sets of data can be extracted. Has anyone tried anything like this?
stuart, I cleaned your code up a little bit. (Untested, but it should work.) Inheriting from the default driver removes the unnecessary methods, and you really just need to return '';
from the export
function.
https://gist.github.com/1910568
No good way of exporting specific columns, at the moment. It wouldn't be too bad to implement (the current code just loops through each field in a section), but it's a little bit of work, so your best bet is to beg kanduvisla. ;)
Also, kanduvisla, any chance you could do some [very basic] error checking on the uploaded file? If the uploaded file is greater than upload_max_filesize
, the import proceeds to step 2 but has no data to map to columns. Took me a while to debug.
In any case, thank you so much! Just imported 20k rows in under 25 minutes. Saved me years of work ;)
I'm trying to use this extension to import data from a legacy CMS for a client and failing. I go through all of the import steps, pick which fields in the CSV to map to which fields in the section, then when I get to the import screen it just sits there on: http://cl.ly/3F1x3U0S2b0i0Z2o1U3V
Doing nothing. Please let me know what other information I can provide you as this data import is reasonably urgent.
Symphony 2.2.5 CSV Import/Export 0.2.3
Just further to the above, looking at the log file I'm seeing the following error:
05 March 2012 14:38 > Warning: GenericErrorHandler 2 - Invalid argument supplied for foreach() in file /Users/liamegan/Dropbox/shared projects/wethecollective/COSSETTE - Taseko/www/html/extensions/importcsv/content/content.index.php on line 186
The works out to the following lines of code:
foreach ($csvTitles as $title) { $ids[] = $_POST['field-' . $i]; $i++; }
Any help that you can provide would be appreciated
Replace the top line with if(is_array($csvTitles)) foreach ($csvTitles as $title)
for a short term fix.
Then file an issue on Github :)
Has anyone written a date driver for Nils Hörrmann's datetime extension? or even a driver for iso dates?
Create an account or sign in to comment.
Its a difficult one to implement as its not predictable to say that each entry takes 'x'kb as it really depends on how the section is structured and the type of fields used. Symphony 2.3 does feature reduced memory usage (around 20% on the default ensemble from memory) so that will help and we're always looking at ways to reduce usage further.