Source Templates – Merge Duplicates #sourcetemplates #sources #merge

2022-03-24 test with #RM8 successful.

RootsMagic has no Merge Source Templates function and it really needs one. When you copy multiple times from one database to another and there are citations involved that use the same custom source template, each time that source template is brought over it is treated as a new source template and the associated source is, too. So now you have multiple identical source templates with no tool to merge them. And you have multiple identical master sources which cannot be merged by AutoMerge because they use different source template records. You can merge them manually but that is painfully tedious given that the identical properties of the source templates should make it possible to automate it.

This script automatically merges custom source templates that have matching (case and space character insensitive 2016-04-22):

  • footnote sentence templates
  • field definitions

Variances in name, the sentence templates for Short Footnote and Bibliography and other fields are ignored which means that source templates that are not quite exact duplicates in all respects will be merged. Unfortunately, it is still sensitive to differences in Hints and Long Hints which are not critical to the merging of source templates.

The first source template (lowest TemplateID) in a set of duplicates will be the last one standing. All sources that once used the duplicates will now use it.

The script does leave a trail of two custom tables that could be cleared out with the addition of DROP TABLE statements added at the end or run separately, identical to the two already in the script.

SourceTemplates-MergeDuplicates.sql Rev 2013-08-07 to correct bug.
Rev 2016-04-22 to make tolerant of differences in case and spacing.

-- SourceTemplates-MergeDuplicates.sql
/*
2013-07-30 Tom Holden ve3meo
2013-08-07 corrected error in Update SourceTable which obtained only the first TemplateID from xLookupSourceTemplateTable
2016-04-22 now tolerates differences in case and space characters in Footnote and FieldDefs
 
Merges custom source templates that have identical
Footnote sentence templates and field definitions.
Ignores differences in Short Footnotes, Bibliography and other fields.
Makes the lowest TemplateID of a set of duplicates the master.
 
It is still sensitive to differences in hints in what would be otherwise identical field definitions that could be merged.
- needs parsing to clear that out.
*/
 
--Create a table of the master custom source templates having duplicate(s)
DROP TABLE IF EXISTS xDupSourceTemplateTable
;
 
CREATE TABLE IF NOT EXISTS xDupSourceTemplateTable
AS
SELECT
   TemplateID
  , Name
  , Description
  , Favorite
  , Category
  , FootnoteCore
  , ShortFootnote
  , Bibliography
  , FieldDefsCore
FROM
(
SELECT COUNT()-1 AS Dupes,
   TemplateID
  , Name
  , Description
  , Favorite
  , Category
  , REPLACE(LOWER(Footnote),' ','') AS FootnoteCore
  , ShortFootnote
  , Bibliography
  , REPLACE(LOWER(FieldDefs),' ','') AS FieldDefsCore
FROM
(SELECT * FROM SourceTemplateTable WHERE TemplateID > 999 ORDER BY TemplateID DESC)
GROUP BY FootnoteCore, FieldDefsCore
)
WHERE Dupes > 0
;
 
-- Create table of matching custom source templates
DROP TABLE IF EXISTS xLookupSourceTemplateIDTable;
CREATE TABLE IF NOT EXISTS xLookupSourceTemplateIDTable
AS
SELECT xD.TemplateID AS MasterID, ST.TemplateID FROM xDupSourceTemplateTable xD
INNER JOIN SourceTemplateTable ST
WHERE xD.FootnoteCore LIKE REPLACE(LOWER(ST.Footnote),' ','')
AND xD.FieldDefsCore LIKE REPLACE(LOWER(ST.FieldDefs),' ','')
AND ST.TemplateID > 999
;
 
-- Revise SourceTable to point to master TemplateID
--EXPLAIN QUERY PLAN
UPDATE SourceTable
SET TemplateID = (SELECT MasterID FROM xLookupSourceTemplateIDTable xL WHERE SourceTable.TemplateID=xL.TemplateID)
WHERE TemplateID IN (SELECT TemplateID FROM xLookupSourceTemplateIDTable)
;
 
-- Delete now unused duplicate Templates
DELETE FROM SourceTemplateTable
WHERE TemplateID IN
(
SELECT TemplateID FROM xLookupSourceTemplateIDTable WHERE TemplateID != MasterID
)
;

Discussions & comments from Wikispaces site


Ksquared333

RMNOCase Error

Ksquared333
02 December 2017 23:08:24

OK. I know I’m being stupid, but I don’t remember how to use RMNOCASE_fake-SQLiteSpy.dll when I’m running SourceTemplates-MergeDuplicates.sql . Where do I load the driver? Also, where do I donate?
Thanks,
Kim Derrick


ve3meo

ve3meo
03 December 2017 02:47:36

This page should answer your question: RMNOCASE+-+faking+it+in+SQLiteSpy

Reports – Point Form Narratives Setup #reports

Rev 2016-04-05: Version 2
Tired of verbose and repetitious sentences in narrative reports and the effort involved in customising and tweaking them so that grammar and prepositions and pronouns were correct, I got the idea some years ago that what I might prefer is a hybrid of the tabular format of the Individual Summary and Family Group Sheet with event/fact notes embedded rather than at the end. Jerry Bryan demonstrated such a concept with his Oct 12, 2015 RootsMagic Forums posting, “Preparing an RM Descendant Narrative Report for a Family Reunion for This Year“. While he did so using largely manual procedures, this SQLite script makes it easy to try out a similar format and then restore your prior format.

SettingsNormalPoint Form
Normal – “Keep facts in same paragraph”Point Form – space character starts fact, 0 Carriage Returns

Note that for all examples of Point Form, the RootsMagic report settings option for paragraphing is set to “Keep facts in same paragraph”.

Although the Point Form is not strictly point form because the facts run together in the same paragraph, it is actually better for fast reading than is the Normal report because there are fact names that catch the eye, the ‘sentences’ are terse and consistent and it is the most efficient in use of space and paper.

DescNarrJohnWoods-Normal0DescNarrJohnWoods-Normal0.pngDescNarrJohnWoods-PointForm0DescNarrJohnWoods-PointForm0.png
Normal – “New paragraph after every fact”Point Form – 2 Carriage Returns start fact

Both of these waste trees!

DescNarrJohnWoods-Normal1DescNarrJohnWoods-Normal1.pngDescNarrJohnWoods-PointForm2DescNarrJohnWoods-PointForm2.png
Normal – “New paragraph after facts with notes”Point Form – 1 Carriage Return starts fact

This Point Form layout is the most attractive of the three or, at least, comes closest to the concept of a hybrid of the Individual Summary and Narrative report.

DescNarrJohnWoods-Normal2DescNarrJohnWoods-Normal2.pngDescNarrJohnWoods-PointForm1DescNarrJohnWoods-PointForm1.png

Features:

  • Backs up key tables or parts thereof to ‘x’ versions which are used by the complementary script to restore back to the previous format.
  • User control* over fact/event sentence heading format: one of plain, bold, italics, underlined.
  • User control* over fact/event sentence lead-in carriage returns.
  • Changes default sentences to point form without [person] variables.
  • Default sentences begin with the fact type name as a heading. This is the name defined in Lists>Fact Type List; the user can edit user-defined fact types in RootsMagic.
  • Alternate Names are labelled by their type, if assigned in the Edit Person screen.
  • Shared events show the role names of the sharers.
  • Adds _Heading event to provide the initial [person] sentence as the first ‘fact’ so each person’s section will start with the person’s name and not be stated as ‘factless’. Version 2 does not do this; instead it uses a Jerry Bryan trick of putting a line feed/carriage return after the [Person] variable for the Birth sentence template, normally the person’s first event. This does require a Birth event as the first event to be output to the narrative report.
  • Adds two trailing CR/LF to the Note field of the last family event (by SortDate) of each couple to force the spouse’s subsection of a person’s report section to start a new paragraph. (V2)

*The script works best with a SQLite manager that supports run-time variables such as SQLite Expert Personal to provide control over the format of fact headings and fact paragraphing. Those that do not support them, such as SQLiteSpy, default to bold and no <CR>s. In either case, a RMNOCASE collation extension is needed. To add the RNMOCASE collation extension, see either:

Version 2
Reports-PointFormNarrativeSetup2.sql
Reports-RestoreAfterPointForm2.sql
Version 1
Reports-PointFormNarrativeSetup.sql
Reports-RestoreAfterPointForm.sql

Use:

  1. Open your database (safer to use a copy) with your SQLite manager.
  2. Load the RMNOCASE extension.
  3. Load the SQLite script Reports-PointFormNarrativeSetup.sql.
  4. Execute the script.
  5. SQLite will throw an error if the database has been previously modified by this script without subsequently running Reports-RestoreAfterPointForm.sql. Load and run the latter and return to step 3.
  6. If prompted for a value for $FactNameFormat_ibun, type in one of the letters: italics, bold, underline, no and OK. Any value other than i, u, n is the same as b.
  7. If prompted for a value for $ParagraphCR_0_1_2, type in one of the numbers 0,1,2 to define the number of Carriage Returns prepended to the sentence. Any value other than 1 or 2 is the same as 0 – a leading space character.
  8. On successful completion the results window will display a status message so reporting.
  9. Open RootsMagic and generate narrative reports. Use the Report Settings Option “Keep fact sentences in same paragraph” to start, especially if you have chosen to have lead-in carriage returns.
  10. You can leave the report preview open and rerun the scripts to switch back to original and to different point form formats. After each execution of a script, you can quickly see its effects from report view with Alt-t to return to Report Settings and <Enter> to regenerate the report with the changes.
  11. You may want to try some other scripts that affect reports after having set up your database for point-form reports:

Maps – Geo-Pedigree, plotting your ancestors’ birth places #googlemaps #geocoding

A RootsMagic Facebook group discussion about pedigree charts colour coded by state or country of birth triggered this idea of plotting ancestors birth places on a map and joining them according to pedigree. This would be a more graphic and informative representation than the standard box chart. This script aids in generating such a map through a web-based front end for Google and other maps.

geo-pedigree.PNG
Example of a geo-pedigree map created through GPSVisualizer.com from the text created by the geo-pedigree script.

The online map is interactive; the screenshot above is not. Clicking a point or line pops out a box with more information. Points or lines can be suppressed or highlighted. A line can be zoomed to its extent with one click. Directional tickmarks indicate the line of descent. The current script plots the five generation pedigree of the root person used by the RootsMagic Set Relationships tool. The waypoint marker for each person’s birthplace follows the person’s color code in RootsMagic; in the example above, the root person was uncolored, paternal ancestors red and maternal lines green. The pedigree lines (or tracks in GPS mapping parlance) follow the colour of the parent. Both tracks and waypoints fade with the number of generations from the root person. The GPS Visualizer input form has numerous controls giving the user choice of the underlying map source and various features in addition to choosing the GPS file to upload. The script queries the RM database to get the needed needed data for each person (geocoded places of birth are essential for it to be meaningful) and writes results in the GPS text format accepted by GPS Visualizer.

I was thinking about other possible improvements to the higgledy-piggledy script I had developed while learning how to use GPSV and what additional information might be included when I discovered that this was not a new idea. There is a very good extension to FamilySearch Family Tree that produces from that database a more powerful geo-pedigree map than I can possibly devise. Perhaps a better use of my time would be to ensure that my ancestors births are accurately recorded in FSFT so that I can rely on the geo-pedigree maps drawn from that data by RootsMapper.com.

RootsMapper.PNG
Screenshot of an interactive pedigree map by RootsMapper.com from data in FamilySearch Family Tree

—TO BE CONTINUED—-

FTM import – restore Event description from Place Details #ftm2014 #ftm2012 #placedetails #events

RootsMagic 7 direct import from Family Tree Maker (not GEDCOM) as of March 2016 imports event descriptions into two different places, depending on the fact type: to the RM event description or to the Place Detail for that event. Some users disagree with that decision and want the FTM event descriptions to remain event descriptions in RM after import. The script provided here can be used on a fresh import to move the content from each event’s Place Detail to the event’s Description field.

The RootsMagician described the import process here:

In FTM, there are fields for date, place, and description. There is not a field for place details.

When RM imports an FTM file, it will place whatever the user entered as a description into the RM description field *IF* that fact type defaults to allowing a description. So, for example, an occupation, residence, etc type fact, RM will import the description into the description field for that fact.

However, FTM lets users enter descriptions for facts like birth, marriage, etc. where it doesn’t make much sense to have a description. So FTM users used that field like RM users used Place Details… the name of the hospital, the street address, etc.

So when RM imports an FTM fact that doesn’t normally allow a description (say birth), it will import that description into the place detail field *IF* the fact actually has a place. If the fact doesn’t have a place then RM will import the description into the description field.

Further discussion in that thread led to this collaboration in which FTM emigre Mike Le Voi packaged a script developed by Tom Holden with instructions for a first-time SQLite user on applying it.

Move RM Place Detail data to Description.pdf Instructions (MLV)
FTMimport-MoveEventPlaceDetail2Description.sql SQLite script (TH)
FTM Import Testing.ftm A FTM database for testing the import process and script (MLV)

N.B. On reusing the revised database, it is probable that it will fail RootsMagic’s File > Database Tools > Test Database Integrity check so simply run the next tool Rebuild Indexes.

Media – Bulk Rename and Relink #media #multimediatable #batch

This page is in response to a question posed in the RootsMagic Forums by member DeliChef in the topic Bulk File Name Change For Hundreds of Family Letters?. He had named the files using the format MM_DD_YYYYPersonName, i.e., the date of the letter followed by name of the sender. To improve the utility of Windows File Explorer, he wanted to change the format to YYYY_MM_DDPersonName so they would sort by name chronologically. They were already linked to the RootsMagic Media Gallery and renaming the files would break the links. The Fix Broken Media Links tool is of no help in this case and manually changing the media item for hundreds of files was too daunting. So he sought help. This page describes a solution that could be adapted to other systematic name revisions.

This solution uses SQLite to query the MultiMediaTable for file names beginning with the MM-DD-YYYY format and generates a set of Windows command line statements that copy these files to a new location with their rearranged names. It also produces a set of SQLite statements that update the MultiMediaTable with the new path and filenames.

N.B., the script was written with “-” instead of “_” as the separator between the date elements and should be edited accordingly to suit your particular needs. Any systematic name pattern can be changed to another pattern.

Here’s an example of the commands produced to relocate and rename the files, copied from the vCmd View created in the database by the script
:

XCOPY "C:SDGoogle Drivegenealogymedia7-25-1900Shaw(Wright)Letitia-PlayCopy.jpg" "C:SDGoogle DrivegenealogymediaNewFolder1900-07-25Shaw(Wright)Letitia-PlayCopy.jpg"
XCOPY "C:SDGoogle Drivegenealogymedia12-01-1949Secombe, Harry (sure).jpg" "C:SDGoogle DrivegenealogymediaNewFolder1949-12-01Secombe, Harry (sure).jpg"
...

and the corresponding SQLite statements to relink the renamed files are copied from the vSQL View created in the database by the script:

BEGIN;
UPDATE MultiMediaTable SET MediaPath = 'C:SDGoogle DrivegenealogymediaNewFolder', MediaFile = '1900-07-25Shaw(Wright)Letitia-PlayCopy.jpg' WHERE MediaID = 18;
UPDATE MultiMediaTable SET MediaPath = 'C:SDGoogle DrivegenealogymediaNewFolder', MediaFile = '1949-12-01Secombe, Harry (sure).jpg' WHERE MediaID = 19;
...
COMMIT;

The XCOPY commands are copied into a DOS batch (.bat) or Windows command line script (.cmd). It is advisable to create the desired destination folder before running this file from the Command Language Interpreter because it may ask whether the target is a folder or a file if the folder does not exist. The correct answer is “file” for which it will create the missing folder(s) in the path.

The SQLite statements are copied into the SQL editor of a SQLite manager such as SQLiteSpy and executed on the database.

The script creates 4 temporary SQLite Views in the RootsMagic database:

  • vOld – holds the MediaID, file path and file name of each file name that matches the MM-DD-YYYY… filename format
  • vNew – holds the MediaID, new path and new file name for each file in vOld
  • vCmd – holds the XCOPY command line statements built from the fields in vOld and vNew
  • vSQL – holds the SQLite UPDATE statements created from vNew

MediaShuffle.sql

Usage

  1. Open your database (best on a copy to be safe) with SQLiteSpy or other SQLite manager that supports the REGEXP function.
  2. Create the NewFolder using File Explorer to match what the script generates (or vice versa).
  3. Load and edit the MediaShuffle.sql script to match your desired new folder name and any differences in the pattern of filenames you wish to match.
  4. Execute the script.
  5. Copy all the Cmd lines from the vCmd View (Ctrl-A, Ctrl-C) in SQLiteSpy to a text file with the .cmd extension.
  6. Open the Windows Command Line Interface (Command prompt) on the folder to which you saved the .cmd file and execute it (type the name of the file and press enter). That should result in the renamed files arriving in your folder.
  7. Copy all the SQLite statements from the vSQL View the script generates into the SQLiteSpy SQL editor and execute against the database. When you open your database in RootsMagic, the Media Gallery will now be pointing to the renamed copies of the files.
  8. When you close the database from SQLiteSpy, the temporary Views will be deleted.

This takes longer to describe than to execute. The slowest part of the process will be the copying of the files. Alternatively, the script could be revised to RENAME files (faster but a little more daring…)

Places – Frequency of Use #places #geocoding

2021-11-22 Compatible with #RM8

See Maps – markers proportional to number of events for an extension of this script to produce results suitable for mapping.

This script produces a list that I think is useful for finding the places that lack an abbreviated name or geographical coordinates and for tackling first the ones having the biggest payoff. For example, a place that is only used once in total or only once for a person does not need a shortened name. On the other hand, a place that is greatly used will add much to a report such as the Place List option “Print events near a place” once it is geo-coded and narratives will sound less repetitive using the abbreviated name. So the list provides two counts:

  • the total number of events in the database that use each place (TotEvents),
  • the highest number of events that any one individual’s events use each place (MaxEvents)

The list includes:

  • PlaceID: the record number in the PlaceTable for the Place in case you want to find it there
  • PersonID: the record number of a Person having the highest number of uses for the Place so that you can inspect the person using RM Explorer; there may be other persons with the same number or less – you could run the inner query to see them all.
  • Place: the full name that is the default output for reports and for the default Place:original variable in sentence templates.
  • Abbrev: the shortened name that is output by the Place:short variable in sentence templates
  • Latitude, Longitude: the geographical coordinates displayed in decimal degrees with North and East being positive values, South and West as negative.
  • Standardized: the name that the Geo-coding function assigns if the place is found in the Gazetteer or the name you input into the Standardized field which might be the contemporary name while the historical name might be entered in the Name field.

Places-Frequency.PNG
Place_Frequency.sql

-- Place_Frequency.sql
/*
2013-03-25 Tom Holden ve3meo
rev 2013-03-25 TotEvents error corrected
rev 2013-03-27 Place:Abbrev corrected to Place:short in comments.
    TotEvents extended to include family events. MaxEvents remains for Indiv only.
    Unused places also listed.
Returns frequency of use for each Place in the database
- total events for each place
- max number of events for any person and a person having that max number
 (family events not counted)
Useful for finding places in need of Abbreviations or Geocoding or Unused
*/
SELECT Places.PlaceID
    ,EventsByPlace AS TotEvents -- total events for place
    ,Events AS MaxEvents -- Max Events for a place by Person (Indiv facts)
    ,PersonID -- Person having the max events for that place
    ,NAME AS Place -- place name used by Place or Place:original in sentence template
    ,Abbrev -- value used by Place:short in sentence template
    ,Latitude / 10000000.0 AS Latitude -- in decimal degrees, North+
    ,Longitude / 10000000.0 Longitude -- in decimal degrees, East+
    ,Normalized AS Standardized -- the Standardized value in the Edit Place screen
FROM PlaceTable Places
LEFT JOIN (
    SELECT PlaceID
        ,PersonID
        ,MAX(Events) AS Events
    FROM (
        -- table of Places for which Persons have events and the number of events for each combinatio
        SELECT PlaceID
            ,OwnerID AS PersonID
            ,COUNT() AS Events
        FROM EventTable
        WHERE OwnerType = 0 -- Individual, not Family, events
        GROUP BY PlaceID
            ,OwnerID -- to aggregate number of events by Place-Person combo
        ORDER BY Events ASC -- to order so that the next GROUP BY PlaceID will extract the highest value of Events for a Place
        )
    GROUP BY PlaceID
    ) AS PersonEvents ON Places.PlaceID = PersonEvents.PlaceID
LEFT JOIN (
    -- table of total events per place
    SELECT PlaceID
        ,COUNT() AS EventsByPlace
    FROM EventTable
    GROUP BY PlaceID
    ) AS AllEvents ON Places.PlaceID = AllEvents.PlaceID
WHERE PlaceType = 0 -- user defined Place; excludes Place Details and Temples
    --GROUP BY Places.PlaceID -- to aggregate TotEvents and extract highest value of MaxEvents for Place-Person combo
ORDER BY MaxEvents DESC -- initial view puts the highest max events first as priority for attention
    ;

Maps – markers proportional to number of events #googlemaps #geocoding

2021-11-22 Compatible with #RM8

An earlier script, Places – Frequency of Use, produced a result set that counted the number of events in each Place in a RootsMagic database. This script builds on that query to generate a comma-delimited data set suitable for uploading to GPSVisualizer.com to generate a map with circle markers that are proportional in size to the number of events.

PlaceFreqGPSVisualizerMap.PNG
Example of a proportional symbol map generated from data extracted by Place_Frequency_GPSVisualizer.

The script produces a result set similar to the screenshot below from SQLiteSpy. Place_Frequency_GPSVisualizer.sql.png
Select any row, then Ctrl-A to select all and Ctrl-C to copy all selected to the Windows Clipboard. You can choose to paste directly into the GPSVisualizer form or into a text editor (NotePad) and save it to a file with the .csv extension for upload and reuse.
GPSVisualizerUploadForm.PNG
Place_Frequency_GPSVisualizer.sql

Spouse Order – Global Sort by Marriage or First-Birth SortDate #spouse

When the person highlighted in Pedigree View or Family View is linked to more than one spouse, RootsMagic 7 lets you select the one to be shown from the dropdown Spouse list. This list also offers a “Rearrange spouses” control which sets the order of the spouses in the dropdown list and the order in which spouses are displayed in the Descendant and Timeline Views, the Edit Person screen and in certain reports. The Rearrange Spouses dialog allows both manual ordering and “Sort by Marriage Date”. This control operates only on the spouses of one person at a time. A wish has been expressed for a global Sort by Marriage Date. Furthermore, an emigre from Family Tree Maker pointed out that it also factors in the birth date of the first child in the case of an unwed couple. These requests may never see the light of day within RootsMagic but here is a script that does the global job in the interim.

Some words of caution. For spouses without marriage events or children, neither the script nor the current RootsMagic function can do anything but sort them to the top of the list. And this script will override sequences arranged manually through RM’s Rearrange Spouses, if there are two or more of the spouses with either a marriage event or child birth event.

Developed using SQLiteSpy and should require no special extensions so it should run with any recent SQLite manager.

SpouseSort2.sql
2016-01-12 Superseded SpouseSort.sql to factor in birth/christen date of first-born children of unwed couples in the absence of a marriage event.

As always, test it on a copy of your database or make a backup first.

Roles-Sort Alphabetically #roles #facttypes

RootsMagic 4-7 (current) list the user-added roles in the Edit Fact Type dialog in the order that they were created, after the builtin roles such as Principal and Witness. For fact types to which many roles have been added, it is desirable that they be sorted alphabetically. While this should easily be done within the program for display purposes and has been requested repeatedly over years, until such an enhancement is provided here is a script that uses a brute force method to rearrange the rows in the data table so that the role names are presented alphabetically.
RoleSortBeforeAfter.png
As far as I can tell, the script is harmless. There is no need to use RootsMagic’s Rebuild Indexes after its execution but it would be advisable to test it on a copy of your database or make a backup before applying it. It was developed using SQLiteSpy but uses no esoteric functions so it should run with any SQLite manager that supports a fake RMNOCASE collation extension..

Caveat: If a future version of RM increases the number of builtin roles to more than 58, every instance of 58 in the script must be revised accordingly else those builtin roles above the RoleID of 58 will be mixed up with a user-added role after a drag’n’drop or GEDCOM transfer to another RootsMagic database..

Tip: If you want the role names to appear in an order sorted by your preference, rename them prefixed by a number, e.g.:

  1. with Parents
  2. with Father
  3. with Grandparents
  4. Other

If you have more than 9 roles, then use double digits:
01 with Parents
02 with Father

RoleSort.sql