VamosaGroups - Comments

Re: Help with programmatic contentDescriptor creation and subseq

Stewart McKee — Sat, 04 Oct 2008 14:16:18 +0100

Hi Ben,

Yes it is possible, and should work fine the way you are doing it with a few tweaks. First thing to check is that the query being used for batch preperation will include your newly created content. If you have not set any batchPrepare.query or batchPrepare.queryLibrary project properties then it will use the default query for the type of project you are using. You don't say what version of the software you are using, but generally for the master project, placeholder and content sub projects it will use "find all html content" which is in the Main Query Library. For the assets sub project it will use the "find all non-html content". These are based on the "Identify Metadata.Content-Type" metadata field. So if you are not populating this, then these queries will not return your new data. You can either supply this to the metadata or create your own query and supply that to batch prepare through the project properties.

thanks,
Stewart.

latest version

Ross MacDonald — Wed, 01 Oct 2008 12:58:19 +0100

Hi Ben,

You do have the latest version for community edition - 2.11. The documentation is for an older version of the product, however the general process will still be the same in 2.11.

Sorry for the confusion.

Ross

V 2.12 resources

humaira — Wed, 01 Oct 2008 12:32:08 +0100

Hi Guys,

I have found one more issue in VCM 2.12, this is to do with resources. If you at any point try to change the Type of the resource, it will wash away all the contents within that resource. I haven’t seen this issue in VCM 2.11.

If you are using VCM 2.12 please make sure you take the copy of the resource content before changing the resource Type.

Huma

29,718 content items stored!

sowens — Fri, 26 Sep 2008 21:54:20 +0100

Thanks Moayyed and Ijonas! Upping the TransactionTimeout value from 300 to 3600 prevented the Timeout from happening and we just got all 30K content items stored in 37 minutes.

I used the same executeSQL method, but Ijonas' suggested rewrite of the method is worth our looking into going forward.

Hmmm, 5 minutes... hmmmm

Ijonas Kisselbach — Fri, 26 Sep 2008 19:12:28 +0100

Hi Steve,

Seem like your task started at 04;10:39 and blew-up at 04:15:36. So that's 5 minutes, or 5*60 seconds, or 300 seconds.Which is roughly the timeout at which point the connection to the DB will get revoked and you end up with a log file such as yours.

So try Moayyed's suggestion and increase the time out from 300 to 3600 (1hr),

Let us know if the problem goes away.

Regards,
Ijonas Kisselbach.
Chief Technology Officer.

Remember page 67 ?

Ijonas Kisselbach — Fri, 26 Sep 2008 19:01:37 +0100

Remember that presentation on Python I did at the Vamosa training day, and how Python's functional programming pieces are really handy constructs... Well guess what ... they're back!!

def executeSQL(sqlString, type, datasourceURL, columnNames, doSomethingInterestingWithRow):
    try:
        # get a reference to JBoss JNDI tree
        context = JNDIUtils.getContextFactory().initialContext
        # lookup the JDBC datasource 
        datasource = context.lookup( datasourceURL )

        if datasource and sqlString != None:
            # the following is standard JDBC code. Get Connection and execute insert
            resultSets = []
            connection = datasource.connection

            if type == 'insert':
                statement = connection.createStatement()
                statement.executeUpdate(sqlString)
            else:
                statement = connection.createStatement()
                results = statement.executeQuery(sqlString)
                while results.next():
                    doSomethingInterestingWithRow( results )
            statement.close()
            connection.close()
    except Exception, e:
        print "Something bad happend %s" % e
    finally:
        if connection != None:
            connection.close()

The problem in the original script is the caching of the whole resultset in RAM, hoping it doesn't blow up, and do some processing outside of the executeSQL() method. So in my revised version, I'm passing in a function-variable called doSomethingInterestingWithRow <-- note: there's no parenthesis used. It can be any method defined elsewhere, which takes a single argument and does something with the result.

This follows the same pattern as the map(F, L) function where F is a reference to a function and L is a list of items to apply the function to.

The main benefit of this approach is separation of concerns, i.e. separating the data gathering from the data processing. Never ever cache whole resultsets in memory.

Regards,
Ijonas Kisselbach.
Chief Technology Officer.

Full Script and Server Log

sowens — Fri, 26 Sep 2008 14:52:20 +0100

Thanks for all the quick responses. I've attached the full script as well as the full error log of the job (in my original post).

I've doubled the TransactionTimeout value in jboss-service.xml from 300 to 600. I will try re-running shortly.

And yes, the contentdescriptors are getting committed to the database

are they committed?

adrianc — Fri, 26 Sep 2008 09:48:13 +0100

When you say it successfully stores 5-10k content descriptors, are they actually committed to the database?

Packages

adrianc — Fri, 26 Sep 2008 09:33:36 +0100

After further experimentation it seems you can wrap your scripts up into their own little package by adding them to a sub-directory of the jythonlib folder simply by adding a __init__.py file into the sub-dir.

The __init__.py file can be simply be empty or you can add to it, see here for examples.

One thing to watch out for, which I could not find any documentation on, is that the sub-directories seem to be limited to 11 characters which had me confused for a while.

So, from the above example... if you dropped simplestuff.py into a sub dir called vcm your import would change to

import vcm.simplestuff

and your code would change to

def enhance( contentDescriptor, content ):
vcm.simplestuff.doSomethingSimple()
vcm.simplestuff.mySimpleClass().doNothing()

ExecuteSQL

mdarugar — Fri, 26 Sep 2008 08:12:13 +0100

Hi Ijonas,

What would you recommend Steve to use in place of executeSQL method ?

Any suggestion you have for him.

Moayyed

Transaction Timeout value in jboss-service.xml

mdarugar — Fri, 26 Sep 2008 08:11:09 +0100

Hi Steve,

I think it may be to do with the transaction timeout in jboss-service.xml file located in 'jboss\server\default\conf' folder. If you look for following

      name="jboss:service=TransactionManager"
      xmbean-dd="resource:xmdesc/TransactionManagerService-xmbean.xml">
      <attribute name="TransactionTimeout">300</attribute>

and change the Transactiontimeout to be more then 300. The time is in seconds and therefore 300 is equal to 5 minutes. It may be due to the fact that the capture is so fast and therefore you may have to increase the transaction timeout. I am not sure if it will resolve the issue, but it is worth trying

Moayyed

Not enough detail

Ijonas Kisselbach — Fri, 26 Sep 2008 07:47:24 +0100

Can you attach the whole script and the whole log file (make sure its a fresh one, as opposed one with 30hrs history) ?

On another note: that executeSQL method is scary! You should never be loading whole resultsets into memory. Bad practice.

Regards,
Ijonas Kisselbach.
Chief Technology Officer.

Thanks Ross, that confirms

adrianc — Sat, 20 Sep 2008 12:04:17 +0100

Thanks Ross, that confirms what I'd figured after realising that image maps (area tags) were not being translated.We added some extra code to handle these separately and all was well after that.

Cheers!

getLinks returns

Ross MacDonald — Thu, 18 Sep 2008 09:11:42 +0100

Hi Adrian,

The getLinks() method will return a HashSet containing all HREF attribute values from 'a' tags.

Another method that can be used is getResourceURLs(), this method will return a HashSet containing all HREF attribute values from 'a' tags and all SRC attribute values from 'img' tags.

You can also use getImages() to return a HashSet of all SRC attribute values from 'img' tags.

If you are going to use the ResourceParser class you should be aware that it will not look at the redirection table for re-writing links when using the rewriteAssetURLs or rewriteImageURLs methods. If you want these redirection links included you should write a script that will populate the translation table with the appropriate values using the redirection table before using the reWrite methods.

Ross

ResourceParser Info???

adrianc — Wed, 17 Sep 2008 16:37:54 +0100

Did you ever find any info on this Ross?

Does this documentation exist?

Specifically I'd like to know what is (and isnt) returned by the getLinks method.

Thanks,

Adrian.