[digiKam-users] removing orphaned xmp files under Linux

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[digiKam-users] removing orphaned xmp files under Linux

Peter Mc Donough
Hi,

I noticed that there are some orphaned xmp-files in my photo folders.

Does anybody know some command line magic for Linux for at least finding
them?

cu
Peter





Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Chris Green
Peter Mc Donough <[hidden email]> wrote:
> Hi,
>
> I noticed that there are some orphaned xmp-files in my photo folders.
>
> Does anybody know some command line magic for Linux for at least finding
> them?
>
Do they have a suffix or at least a recognisble pattern in their name?
If so it's dead easy using the 'find' command:-

    find <name of top level directory> -name '*.xmp'

If you want to remove the files as well:-

    find <name of top level directory> -name '*.xmp' -exec rm {} \;

--
Chris Green
·

Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

digikam-2
On Wed, 7 Mar 2018 21:35:30 +0000
Chris Green <[hidden email]> wrote:

> Do they have a suffix or at least a recognisble pattern in their
> name? If so it's dead easy using the 'find' command:-
>
>     find <name of top level directory> -name '*.xmp'
>
> If you want to remove the files as well:-
>
>     find <name of top level directory> -name '*.xmp' -exec rm {} \;

I don't think so. It finds ALL the files that have an .xmp
extension, including all the files that have a raw counterpart.

Doing the rm will delete all the xmp files

> find ~/darktable/ -name '*.xmp'|less

Here is an xmp from the find:

/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp

> ls /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363*

/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2
/home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp



--
sknahT

vyS
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
I'm not a bash expert whatsoever, but I think this could work, I based it on the script found here: https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extension Run it from your picture directory. It will scan and delete all orphaned xmp files. Right now it would only show which files would be deleted. Uncomment the second to last line for actually deleting the orphaned files, or the last one to rename to orphaned files with a ".old" extension (so you can find and delete them later). I tried it with a dir with fake contents, and it seems to work. But please, use it at your own risk.
#!/bin/bash
shopt -s nullglob extglob;

#dir=~/mypictures
#cd $dir

# explore dirs recursively, finding .xmp files
find . -type f -name "*.xmp" | while read file
do
    #echo "$file"
    candidates=("${file%.*}"@() "${file%.*}".{cr2}@());
    [[ ${#candidates[@]} -eq 0 ]] && echo "deleted orphaned file $file"
    #[[ ${#candidates[@]} -eq 0 ]] && rm "$file" 
    #[[ ${#candidates[@]} -eq 0 ]] && mv "${file}" "${file}.old"
done



Sent from the digikam-users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
Am 08.03.2018 um 03:30 schrieb woenx:

> I'm not a bash expert whatsoever, but I think this could work, I based it on
> the script found here:
> https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
> it from your picture directory. It will scan and delete all orphaned xmp
> files. Right now it would only show which files would be deleted. Uncomment
> the second to last line for actually deleting the orphaned files, or the
> last one to rename to orphaned files with a ".old" extension (so you can
> find and delete them later).I tried it with a dir with fake contents, and it
> seems to work. But please, use it at your own risk.
>

Whow! Here is the copy from:

#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#
# --------------------

  I'll test it later with "mv" instead of "rm"

Thanks
Peter
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
Am 08.03.2018 um 10:00 schrieb Peter Mc Donough:
> Am 08.03.2018 um 03:30 schrieb woenx:
>> I'm not a bash expert whatsoever, but I think this could work, I based
>> it on
...

Tested, but my Kubuntu's Bash doesn't like "shopt", error message below.
I supose a shell option. It probably has to be activated somehow:

Any ideas
Peter

-----------------
Error Message:
/home/peter/foto_xmp_ex.sh: 5: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 12: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")

> shopt -s nullglob extglob;
>
> # Get all sidecar files
> for file in *.{xmp,pts,pp3}
> do
>    # Generate all permutations of filenames that it may belong to,
>    # and let globbing delete the ones that don't exist
>    candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());
>
>    # If none exist, the file can be deleted
>    [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
> done
> #

Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
Am 08.03.2018 um 12:28 schrieb Peter Mc Donough:

I better add the complete shell script

cu
Peter


#!/bin/bash
# 20180308 - entfernen ueberfluessiger "xmp/pts/pp3" -Dateien
#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#

Error message:
peter@kubu2-lux:~/temp/oldies_mcd/2$ sh  ~/foto_xmp_ex.sh
/home/peter/foto_xmp_ex.sh: 5: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 12: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")
peter@kubu2-lux:~/temp/oldies_mcd/2$
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
Did you keep the "#!/bin/bash " at the beggining of the script? I think the shopt command only works in bash. What happens if you type it directly on the terminal? By the way, the difference between the script on stackoverflow and the one I pasted, is that mine scans directories recursively.

Sent from the digikam-users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Chris Green
In reply to this post by digikam-2
[hidden email] wrote:

> On Wed, 7 Mar 2018 21:35:30 +0000
> Chris Green <[hidden email]> wrote:
>
> > Do they have a suffix or at least a recognisble pattern in their
> > name? If so it's dead easy using the 'find' command:-
> >
> >     find <name of top level directory> -name '*.xmp'
> >
> > If you want to remove the files as well:-
> >
> >     find <name of top level directory> -name '*.xmp' -exec rm {} \;
>
> I don't think so. It finds ALL the files that have an .xmp
> extension, including all the files that have a raw counterpart.
>
> Doing the rm will delete all the xmp files
>
> > find ~/darktable/ -name '*.xmp'|less
>
> Here is an xmp from the find:
>
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp
>
> > ls /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363*
>
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2
> /home/froggy/darktable/people/photographers/photographers-2014/photographers-20141107-0363.cr2.xmp
>

OK, that's why I asked "Do they have a suffix...", I meant the orphan
ones.

So you'll have to write a simple (bash?) script that uses find (as
above) to find all the xmp files and then removes only the orphans.

The logic would be something like:-

    find an .xmp file
    strip off the .xmp (use basename)
    find the 'parent' file (the one with the .xmp removed)
    if the above fails then remove the .xmp file

--
Chris Green
·

Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
This is exactly the script I pasted above does. I recursively scans for .xmp
files and checks if there's a matching cr2. If there isn't, the xmp is
deleted.

I think the "shopt" commands requires a bash console. Did you put the
"#!/bin/bash" at the beginning of the script? What happens if you just run
the "shopt -s nullglob extglob" in a console?



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

digikam-2
In reply to this post by Peter Mc Donough
On Thu, 8 Mar 2018 10:00:05 +0100
Peter Mc Donough <[hidden email]> wrote:

> # shopt -s nullglob extglob;
>
> # Get all sidecar files
> for file in *.{xmp,pts,pp3}
> do
>    # Generate all permutations of filenames that it may belong to,
>    # and let globbing delete the ones that don't exist
>    candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf}@());
>
>    # If none exist, the file can be deleted
>    [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
> done
> #
> # --------------------
>
>   I'll test it later with "mv" instead of "rm"

Watch for this script, there is a "major weakness".

Some softwares (like darktable) create multiple xmp files (xxx_01,
xxx_02, xxx-1, xxx-2...) for the same raw file as virtual copies and
the script will also delete them.

--
sknahT

vyS
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
In reply to this post by woenx
Am 08.03.2018 um 17:32 schrieb woenx:
> This is exactly the script I pasted above does. I recursively scans for .xmp
> files and checks if there's a matching cr2. If there isn't, the xmp is
> deleted.
>
> I think the "shopt" commands requires a bash console. Did you put the
> "#!/bin/bash" at the beginning of the script? What happens if you just run
> the "shopt -s nullglob extglob" in a console?
>

just
shopt -s nullglob extglob

as user and as root

No output??

cu
Peter
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
Yep, that's right, no output means that the command was executed
successfully.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
Am 08.03.2018 um 21:43 schrieb woenx:
> Yep, that's right, no output means that the command was executed
> successfully.
>

Where is my mistake?

I create an test-folder im my home-directory,
copy three raw-files including their xmp-files,
remove one raw-file to get one ophaned xmp-file.

I added ls -l to the script and "rw2" to "candidates"

If I run the script in that folder, the output should show, at the end,
the orphaned xmp-file but I get only the list of all five files and
still the error message below.
The error points to  one "(" too many or too few in the line with the
candidates comparisons.


#!/bin/bash
# 20180308 - entfernen ueberfluessiger "xmp/pts/pp3" -Dateien
#
https://stackoverflow.com/questions/48757748/removing-orphaned-sidecar-files-by-extensionRun
#
ls -l
#
shopt -s nullglob extglob;

# Get all sidecar files
for file in *.{xmp,pts,pp3}
do
   # Generate all permutations of filenames that it may belong to,
   # and let globbing delete the ones that don't exist
   candidates=("${file%.*}"@() "${file%.*}".{nef,raf,orf,rw2}@());

   # If none exist, the file can be deleted
   [[ ${#candidates[@]} -eq 0 ]] && echo rm "$file"
done
#

-------------
error message
-------------

peter@kubu2-lux:~/test$ sh ~/foto_xmp_ex.sh
insgesamt 38332
-rw-r--r-- 1 peter peter 19478572 Apr  9  2015
20150314_155955_AIDAluna_orgP.rw2
-rw-r--r-- 1 peter peter     6283 Jan 17 17:42
20150314_155955_AIDAluna_orgP.rw2.xmp
-rw-r--r-- 1 peter peter 19744364 Apr  9  2015
20150314_162153_AIDAluna_orgP.rw2
-rw-r--r-- 1 peter peter     5923 Jan 17 17:42
20150314_162153_AIDAluna_orgP.rw2.xmp
-rw-r--r-- 1 peter peter     5924 Jan 17 17:42
20150314_163131_AIDAluna_orgP.rw2.xmp
/home/peter/foto_xmp_ex.sh: 7: /home/peter/foto_xmp_ex.sh: shopt: not found
/home/peter/foto_xmp_ex.sh: 14: /home/peter/foto_xmp_ex.sh: Syntax
error: "(" unexpected (expecting "done")
peter@kubu2-lux:~/test$

Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
This happens if the /shopt -s nullglob extglob;/ command is not recognised in
the script. I guess that for some reason, the script is not using bash.

Include this line at the beginning of the script, to identify which shell
you're using:



For instance, if your script uses /zsh /instead of /bash/, the command
should be:





--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
Dammit, the message lost all its formatting.

Let me repeat it:

This happens if the shopt -s nullglob extglob; command is not recognised in the script. I guess that for some reason, the script is not using bash.

Include this line at the beginning of the script, to identify which shell you're using:
ps -p $$
For instance, if your script uses zsh instead of bash, the command should be:
setopt nullglob extglob;


Sent from the digikam-users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
In reply to this post by woenx
Am 09.03.2018 um 00:52 schrieb woenx:
> This happens if the /shopt -s nullglob extglob;/ command is not recognised in
> the script. I guess that for some reason, the script is not using bash.
>
> Include this line at the beginning of the script, to identify which shell
> you're using:

??

> For instance, if your script uses /zsh /instead of /bash/, the command
> should be:

Sorry, what line and what command ?

Peter
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

woenx
Argg, I'm sorry, the message lost its format and misses half the content. I
resent it again but now it says that it's "pending moderation", so I guess
it'll take a while to be published.



--
Sent from: http://digikam.1695700.n4.nabble.com/digikam-users-f1735189.html
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
Am 09.03.2018 um 01:01 schrieb woenx:
> Argg, I'm sorry, the message lost its format and misses half the content. I
> resent it again but now it says that it's "pending moderation", so I guess
> it'll take a while to be published.
>

No problem, Google helped

the shell is bash.

peter@kubu2-lux:~$ ps -p $$
   PID TTY          TIME CMD
  9108 pts/1    00:00:00 bash
peter@kubu2-lux:~$


Peter
Reply | Threaded
Open this post in threaded view
|

Re: removing orphaned xmp files under Linux

Peter Mc Donough
In reply to this post by woenx
Am 09.03.2018 um 00:57 schrieb woenx:

> Dammit, the message lost all its formatting.
>
> Let me repeat it:
>
> This happens if the /shopt -s nullglob extglob;/ command is not recognised
> inthe script. I guess that for some reason, the script is not using bash.
>
> Include this line at the beginning of the script, to identify which
> shellyou're using:For instance, if your script uses /zsh /instead of /bash/,
> the commandshould be:
>

Still, lost in transmission!

but the shell is

peter@kubu2-lux:~$ echo $0
/bin/bash
peter@kubu2-lux:~$

and I included in the script

#!/bin/bash
# and
ls -l
# which listed the files.

cu
Peter
12