[EP-tech] Re: Thousands of old eprints repropagated via OAI after epadmin redo_thumbnails &co.

Sebastien Francois sf2 at ecs.soton.ac.uk
Wed Apr 9 17:46:07 BST 2014


On 09/04/14 10:31, Florian Heß wrote:
> Am 08.04.2014 11:57, schrieb Sebastien Francois:
>> - recommit: by definition, this action should touch lastmod
> Hi Sebastien,
> I am afraid, I disagree here partly. Recommits should touch lastmod only
> *if* there are dirty substantial = user-editable metadata columns. This
> admittedly is difficult to decide by epadmin recommit tool as the
> changes often have taken place directly in advance, bypassing the API,
> for this I assume is the main purpose of that tool. Hence, what about
> --non-volatile-change alias --no-touch-lastmod switches (and/or their
> respective positive counterparts) to epadmin recommit and alike?

Fair point!...

So when someone runs epadmin recommit, the script forces the recommit 
($dataobj->commit( 1 )) which *always* writes data to the database, 
regardless of whether the data-obj was modified during the re-commit.

So if I understand correctly your case-studies, it seems like you need:

1- epadmin recommit
2- epadmin force-recommit

v1 could do $dataobj->commit() -> no change in the dataobj = no DB 
write, no update of lastmod, revision, history. Change in the dataobj => 
lastmod, revision, history updated

v2 could do $dataobj->commit( 1 ) -> same as the current behaviour

> Look, there are so many actions an eprint commit trigger (e.g.
> /cfg.d/eprint_fields_automatic.pl) might include that you developers
> possibly cannot forsee, that you maybe would [not] consider an
> anti-conception feature misuse, and that might need a "recommit"
> sometimes e.g. when code just added requires all older eprints to be
> reprocessed. Touching lastmod no matter if a specific data object meets
> any seldomly occurring conditions for a given action, can result in
> problems. A guy from China had problems accessing our OAI server after -
> and maybe just because - we regenerated the thumbnails, thus potentially
> making they swallow half of HeiDOK.

Shouldn't touch lastmod if regenerating thumbnails, we definitely agree 
on this.

But for other cases, if the metadata is updated then lastmod must be 
updated. And the OAI protocol is consistent with this behaviour:

A repository*must*update the datestamp of a record if a change occurs, 
the result of which would be a change to themetadata part 
the XML-encoding of the record. Such changes include, but are not 
limited to, changes to the metadata of the record, changes to the 
metadata format of the record, introduction of a new metadata format, 
termination of support for a metadata format, etc.

Volatile changes shouldn't cause an update of the lastmod datestamp (by 
definition... it's a volatile change, not a content/metadata change). 
This is how EPrints works (but I agree stuff must be fine-tuned... hence 
our discussion :-)).

So what you cannot have (if you want to respect OAI) is updating some 
item's metadata and not update the lastmod field (you can of course find 
ways to achieve this result but not OAI-friendly).

> Some more info on my OAI-harvesting aggregator scenario so you
> understand my problem:
> The aggregator database is kept small by dropping items that have not
> been modified for more than 100 days. Practically, epadmin recommit is
> therefore a superb tool to make our "new media" service advertise rather
> old if not obsolete stuff. According to OAI specification (as is how I
> remember once having read), OAI-compliant repositories should bear in
> mind harvesters not mirroring all of a data provider. This includes in
> my eyes that the data provider should repropagate records with some
> caution in order to not irritate "bleeding edge stuff" harvesters. Sure,
> one can still argue that those are better off considering dc:date more,
> but this is not always an appropriate filtering criterion.

So... given the conversations we've had so far - would the suggested 
changes (the "don't touch lastmod if no metadata change has occurred") 
be compatible with what you're trying to achieve?

Thanks for keeping the conversation up. Once we're both happy, let's 
propagate this to github.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.ecs.soton.ac.uk/pipermail/eprints-tech/attachments/20140409/6e4d55dd/attachment.html 

More information about the Eprints-tech mailing list