FALCON-2112 Set property value to set map memory for replication and retention #264
Conversation
… retention. Removed extra unwanted line
sandeepSamudrala
changed the title
sandeepSamudrala
changed the title
sandeepSamudrala
changed the title
|
@sandeepSamudrala Can't parameters for map task memory be set in the Feed entity properties section. |
|
@peeyushb : Yes. They can be set. This way gives it to define globally for all feeds and not each feed has to specify it. |
|
Each feed entity must have it's own calculated requirement of memory for retention and replication job, so I am more of the opinion let each feed entity provide memory requirements by specifying in properties rather setting it globally across for all feeds. Thoughts please. |
|
In addition, "oozie.launcher.mapreduce.map.memory.mb" and "mapreduce.map.memory.mb" is not just one memory parameter for Hadoop jobs. There are bunch of other memory parameters as well and those parameters values are set in accordance with each other. |
|
@peeyushb : Even with this patch, user can specify his own requirements in the feed properties and they have precedence over global settings. In an organization running large clusters, it would be difficult to update all the feeds and so this provides leisure to just touch the feeds and the new settings will get kicked. |
| <property> | ||
| <name>oozie.launcher.oozie.libpath</name> | ||
| <value>${wf:conf("falcon.libpath")}</value> | ||
| </property> |
There was a problem hiding this comment.
If you are setting the launcher memory for others, better be consistent and set it here too.
There was a problem hiding this comment.
Ack. I will put launcher memory here too.
|
@peeyushb : On coming to other parameters. Yes I agree, there are bunch of other parameters which work in accordance with the above settings. But coming to feed replication and retention there is no MR job, they are simple java action to delete data and distcp job to copy the data, which won't require many other settings. These properties are set only to feeds and not processes. |
|
Please also update documentation. The build has failed, but, I guess the ITs that have failed are un-related. |
…l eviction actions and updated the documentation
|
+1 from my end. @peeyushb, are all your comments addressed? |
|
@sandeepSamudrala Just setting this particular memory parameter alone and that too globally for all the Falcon feed entity scheduled MR jobs, many of the times it will not help, if other memory parameters are not in accordance and user has not defined anything in feed entity to override the global settings. Also, why you think that update functionality for feed entity can't be leveraged to set the relevant memory through properties for the required and particular feed entity MR jobs. As definitely not all the feed entities in Falcon may require this. If we do comparision with Oozie, if any particular workflow has different memory requirement they just specify the value to the required memory parameters for that particular workflow only. Moreover, feed replication and retention run as MR jobs. |
|
Hi @sandeepSamudrala - setting map memory mb alone is not enough. Since the minimum allocation will be determined by yarn.scheduler.minimum-allocation-mb and yarn.scheduler.maximum-allocation-mb you may indeed get more than 512mb. Also, even if you changed the minimum allocation to 512m (which should not be the default as you have), then you need to make sure that the mapreduce.map.java.opts has -Xmx set less than that with allowance for the JVM requirements. |
|
@vrangan : Thanks for pointing the same. I had realized what other effects it could introduce. I had made the changes accordingly for the same to accommodate mapreduce.map.memory.mb I will test the updated patch and update the pull request. |
…nfigs for feed workflows
| the maximum number of maps used during replication. "mapBandwidth" represents the bandwidth in MB/s | ||
| used by each mapper during replication. "overwrite" represents overwrite destination during replication. | ||
| used by each mapper during replication. | ||
| "mapMemory" represents the mapreduce map memory in mb to be specified to the respective replication and retention jobs. |
There was a problem hiding this comment.
nit: just to keep it consistent either start it from the same line or move every property name to a separate line.
Right now mapBandwith is in one line and rest of setting is in form of paragraph.
There was a problem hiding this comment.
Ack,will make the changes accordingly.
5 participants
No description provided.