The problem of finding an optimum of a multivariate function described as a sum of potentials over (small) subsets of variables is one of fundamental interest both in probabilistic inference and other fields. In this thesis we present a cycle-cutset driven stochastic local search algorithm which approximates the optimum of sums of unary and binary potentials, called Belief Propagation Local Search or BPLS. We evaluate empirically the effects of different components of BPLS on its performance and present the results of extensive and comprehensive experiments conducted on several problem sets. We further suggest several novel heuristics designed to improve the exploration of the search space based on previous results and more detailed analysis of the energy function. Some of the heuristics and observations made are general, and may be applied in other local search algorithms and in other contexts. We present experimental results supporting the contribution of these heuristics. Finally, we compare the performance of the leading variants of BPLS to the state-of-the-art GLS+ and against a hybrid. We show that in general the performance of BPLS is on-par with GLS+ and that it significantly outperforms GLS+ on the CSP problem set. Our results are significant because for the past decade, GLS+ is well established as the best stochastic local search for MPE. Moreover, BPLS reaches strong local optima in the limit (conditionally optimal on every tree), and as such provides an effective and convergent algorithm alternative to min-sum BP schemes. In the second part of this thesis we explore the notion of tree inducing cycle-cutset and present novel theoretical results. We prove that in grids of any size there exists a minimal cycle-cutset whose complement induces a single connected tree. More generally, any cycle-cutset in a grid can be transformed to a tree-inducing cycle-cutset, no bigger than the original one in a series of steps from a given cutset to another. We use this result to improve the known lower bounds on the size of a minimal cycle-cutset in certain cases of grids, thus equating the lower bound to the known upper bound.